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ABSTRACT 


Homeland Security Presidential Directive #8 led to the establishment of 
the National Exercise Program and the Top Officials exercise series to test and 
evaluate first response agency integration and effectiveness. The last TOPOFF 
exercise cost $16M and involved over 10,000 people, but did not effectively 
leverage simulation techniques to make efficient use of resources. 

This research adapts an existing organizational learning process, 
integrating low- and high resolution simulation to provide decision support. This 
process led to the development of a multi-agent simulation methodology for 
emergency first response, specifically applied to analyze a notional vehicle bomb 
attack during a festival in the Baltimore Inner Harbor. 

This simulation demonstrates the potential benefits of low resolution 
simulation, using efficient experimental design and high-performance computing. 
Combined, these two ideas result in examining a 48-dimensional response 
surface and using over 156 CPU centuries of computer time. All experiments 
were completed in less than three weeks. 

The analysis of this data set provided insight into several areas, including 
the importance of standing operating procedures in the early moments of a crisis. 
Analysis showed that effective procedures may even be more important than the 


effectiveness of communications devices early in a first response operation. 
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EXECUTIVE SUMMARY 


Large-scale disasters can quickly overwhelm the capabilities of state and 
local governments. An effective response in these situations results from 
integrating state and local agencies with their federal counterparts, thus enabling 
the flow of needed resources and knowledge. Toward this end, a 
Presidential Directive was issued as part of a plan to prepare for and mitigate the 
effects of crisis events. This directive led to the establishment of the National 
Exercise Program (NEP). National-level exercises, such as those that comprise 
the NEP, test and evaluate federal, state, and local (FSL) integration and 
readiness to confront a manmade or natural disaster. 

Top Officials (TOPOFF) exercises are the foundation of the NEP. These 
large-scale exercises involve participation from all levels of governmental and 
nongovernmental agencies inside and outside the United States. These 
exercises are currently being planned and executed with very little consideration 
given to the value of simulation as a preparation tool. 

Simulation is a widely used decision support tool because it allows staffs 
and decision makers to explore given problems in ways that are otherwise 
impractical (e.g., due to resources needed) or impossible (e.g., running an 
exercise with thousands of parameter permutations). The Department of 
Homeland Security (DHS) recognizes the value of simulation and reviews 
simulation models for applicability before each TOPOFF planning process 
begins. As yet, DHS has not found the right tool for the job. 

Figure ES1 demonstrates an organizational learning process adapted 
from a methodology developed by the U.S. Army Training and Doctrine 
Command Analysis Center — Monterey (TRAC-MTRY); this is the right tool for 
the job. This methodology is an iterative process that uses a quick turnaround, 
low resolution model to provide initial insights into a given problem. Those 
insights are used in the execution of a high resolution simulation, such as a 


wargame. As with any high resolution simulation, wargaming results can be 


Xxi 


actionable results; that is, the decision maker can use these results to finalize the 
plan that was wargamed. However, the decision maker can also decide to adjust 
the low resolution simulation and iterate the process until they obtain 


satisfactory results. 
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Figure ES1. Adaptation of TRAC-MTRY's Learning Methodology 



































The process established in this research expands the TRAC-MTRY 
methodology, which includes a general flow that uses the power of simulation to 
train a given audience. This research shows a specific process by which 
low resolution and high resolution simulation can be used together to help 
organizations prepare for a TOPOFF exercise, or any other large-scale 
training exercise. 

High-resolution simulations, such as wargames, are established 
decision support tools. TRAC-White Sands Missile Range (WSMR) developed a 
model to facilitate the execution of wargames in a first-response setting. The 
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Emergency Preparedness incident Command System (EPiCS) was used in 
February 2006 to simulate emergency first response to a bomb attack in 
Baltimore’s Inner Harbor area, during a festival. To assess and demonstrate the 
potential of an agile, low resolution simulation in this methodology, a multi-agent 
simulation (MAS) was developed to simulate the same vignette. The simulation 
involves a small terrorist cell that detonates a car bomb, then works to further 
incite panic, while a gunman lies in wait for first responders to attack. Police, fire, 
and medical personnel respond to the bomb blast area, in which walking 
wounded and stretcher wounded civilians are panicking. First responders have 
the following priorities of work, in which police, Emergency Medical Technicians 
(EMTs), and firemen will: 


° Stabilize wounded civilians 

® Restore calm in the area 

e Eliminate further threats 

° Maintain safety of first responders 


Figure ES2 is an_ illustrated screen shot of the emergency 
first response MAS. 
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Figure ES2. Screen Shot of Emergency First Response MAS 
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In the analysis of the first responder effectiveness, this research involved 
exploring a 48-dimensional space to gain an understanding of the complex 
relationships involved in this problem. This exploration required an efficient 
design of experiments. A traditional gridded design would have resulted in 
experimental runs that lasted 116 trillion times the current age of the universe. 
The design of experiments developed for this research uses both the flexibility of 
Flexible Random Latin Hypercube (FRLH) sampling, and the space filling nature 
of Nearly Orthogonal Latin Hypercube (NOLH) sampling. The use of these 
efficient designs, in conjunction with use of the supercomputers at the 
Maui High Performance Computing Center (MHPCC), resulted in the use of 
137,277,343 CPU hours, or 156 CPU centuries. The data set analyzed for this 
research is nearly 52,000 rows by more than 5,300 columns. 

The analysis of the data from this model suggest: 

° Overwhelmingly, the most important factor in achieving success in 

crisis mitigation is the effectiveness of the police in taking positive 


control of the crowd, exerting calming influence, and providing 
direction. 


° If a police force is not well trained, and therefore not very effective, 
the officers may achieve greater success by being less persistent 
with individuals; that is, by spreading their influence more broadly. 


° Well established, well executed standing operating procedures 
(SOPs) may play a more important role in first response operations 
than interagency communication. 


e There may be a level of diminishing returns for first responder 
training; that is, a person can be only so trained. After that level is 
reached, it may be more effective to leverage resources elsewhere. 


Results of the data analysis are not meant to directly apply to actual 
emergency response techniques, or specifically to the City of Baltimore. This 
model does not include the actual force structure and SOPs from Baltimore, but 
data adapted from the February 2006 EPiCS run. This research is a proof of 
concept to show that it is possible to quickly and credibly model emergency 
first response with a MAS, and the data analysis from such a credible, verified, 
and calibrated model will be useful and insightful. 
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The single most important result of this research comes not from the data 
analysis, but from the developed methodologies. Simulation is a decision 
support technique that is relevant to emergency preparedness, especially to an 
exercise program the size and complexity of the TOPOFF program. The 
organizational learning technique discussed herein and the incorporation of MAS 
in emergency first response simulation can help train first response organizations 


more effectively, resulting in better crisis mitigation and lives saved. 
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I. INTRODUCTION 


Our Federal, State, and local governments would ensure that all 
response personnel and organizations—including law enforcement, 
military, emergency response, health care, public works, and 
environmental communities—are properly equipped, trained, and 
exercised to respond to all terrorist threats and attacks in the 
United States. ' 


National Strategy for Homeland Security 
July 2002 


A. NATIONAL EXERCISE PROGRAM BACKGROUND 


The President issued Homeland Security Presidential Directive Number 8 
(HSPD-8, Subject: National Preparedness) as part of a plan to prepare for and 
mitigate the effects of crisis events. In HSPD-8, the President directed the 
Secretary of the Department of Homeland Security (DHS) to 

.. . establish a national program and a multi-year planning system 

to conduct homeland security preparedness-related exercises that 

reinforces identified training standards, provides for evaluation of 

readiness, and supports the national preparedness goal.® 

This presidential directive led to the establishment of the National Exercise 
Program (NEP). National-level exercises, such as those that comprise the NEP, 
evaluate federal, state, and local (FSL) integration and test “collective 
preparedness, interoperability, and collaboration across all levels of government 
and the private sector.” 

State and local preparedness is necessary, but insufficient, for effective 
disaster response, as large-scale disasters can quickly overwhelm the 





"Homeland Security Exercise and Evaluation Program, Volume I: Overview and Doctrine, 
revised May 2004, p. 4. Retrieved on 26 January 2006 from the World Wide Web at 


http://www.ojp.usdoj.gov/odp 


*“"4qomeland Security Presidential Directive #8.” Retrieved on 5 December 2005 from the 
World Wide Web at http:/Avww.whitehouse.gov/news/releases/2003/12/20031217-6.html 


3Ibid. 


capabilities of state and local governments. Effective response in these 
situations results from integrating state and local agencies with their federal 
counterparts, thus enabling the flow of needed resources and knowledge. 

The NEP is one component of the DHS’s endeavor to achieve the 
National Preparedness Goal (NPG), established by HSPD-8. The NPG 
“envisions a national system in which all agencies have the capabilities they 
need to effectively communicate and coordinate resources.”* The following 
seven measurable priorities quantify the level of FSL preparedness and 
achievement of the NPG. 

Overarching Priorities. | Three NPG priorities contribute to the 


development of several capabilities: 


e Implement the National Incident Management System (NIMS) and 
National Response Plan (NRP) 

° Expand regional collaboration 

e Implement the Interim National Infrastructure Protection Plan 


Capability-Specific Priorities. _ Four NPG priorities are capability- 
specific, providing specific means that the nation needs most: 


° Strengthen information sharing and collaboration capabilities 

e Strengthen interoperable communications capabilities 

e Strengthen chemical, biological, radiological, nuclear, and explosive 
(CBRNE) detection, response, and decontamination capabilities 

e Strengthen medical surge and prophylaxis capabilities to enable 
emergency-ready public health and medical facilities across 
the nation® 


Four of the seven NPG priorities address the need for efficient and 
effective command and control, communication, and resource sharing in crisis 
events. The NEP established the Top Officials (TOPOFF) exercise as their 

“Fact Sheet: The National Priorities,” Department of Homeland Security. Retrieved on 


5 December 2005 from the World Wide Web at_http://www.ojp.usdoj.gov/odp/docs/ 
Priorities 041305.pdf 


“Ibid. 


cornerstone event to assess the progress of DHS in meeting its 
NPG priorities. 

TOPOFF is a biennial series of exercises mandated by Congress to 
strengthen the nation’s crisis response.° These exercises have grown in size 
and scope since the first one in 2000. TOPOFF 2000 was a $3.5 million national 
event, incorporating “numerous” FSL agencies.’ TOPOFF 3, the most recent 
exercise, was a $16 million international event with over 200 FSL agencies and 
10,000 participants.2 The resources dedicated to this exercise series help 
demonstrate the importance of this program and its role in developing crisis 
response capabilities. This research will support the TOPOFF program by 
helping the exercise planning and execution process mature to more efficiently 
and effectively meet the NPG. 


B. SIMULATION AS A TOPOFF FACILITATOR 


Simulation is one of the most widely used operations research and 
management science techniques, if not the most widely used.’ 


Law and Kelton, 2000 


Simulation is a widely used decision support tool because it enables an 
organization’s staff to efficiently examine problems that would otherwise be 
expensive to analyze. The staff presents its recommendation to the 
decision maker based on analysis of the simulation. For example, suppose a 


°Department of Homeland Security, “National Exercise Program Overview,” Office for 
Domestic Preparedness. Retrieved on 1 December 2005 from the World Wide Web at 
http://www. ojp.usdoj.gov/odp/exercises.htm 


’U.S. National Response Team, “Exercise TOPOFF 2000 and National Capital Region After 
Action Report,” 2001. Retrieved on 5 December 2005 from the World Wide Web at 
http://www.nrt.org/Production/NRT/NRTWeb.nsf/PagesByLevelCat/Level3TOPOFF? 
Opendocument 


’Department of Homeland Security, Press Room, “TOPOFF 3 Frequently Asked Questions.” 
Retrieved on 2 March 2006 from the World Wide Web = at_ hittp:/Awww.dhs.gov/ 
dhspublic/interapp/editorial/editorial 0603.xml. 


*Averill M. Law, and W. David Kelton, Simulation Modeling and Analysis, 3 Edition, 2000, 
p. 2. 


manufacturing company is considering the construction of an extension to one of 
its plants. The company will not build the extension, track change in profit, then 
tear down the extension if it results in a net loss. However, the company may 
conduct a form of simulation to examine the likely change in profit based on the 
decision to build. Using this analysis, a recommendation is made to the 
decision maker to build or not to build.'° 

Simulation is also a useful decision support technique for analyzing the 
execution of contingency plans. The U.S. military frequently wargames its 
military plans before the execution of the plan. Wargaming is a “human-in-the- 
loop” simulation. Human-in-the-loop means that people are actually involved, 
making decisions, throughout the simulation. In wargaming, a plan is analyzed 
by examining an event in terms of a side’s action, another side’s counteraction, 
and the first side’s subsequent counteraction, and so on. Through analyzing 
both the results of the simulation and the process by which those results were 
achieved, a staff can recommend a plan to the decision maker for approval. 

In addition to its usefulness in determining the allocation of resources and 
analyzing contingency planning, simulations are good training tools. 
Human-in-the-loop simulations, such as wargames, provide the target audience 
with a chance to react to situations they may encounter when the actual event 
occurs. Computer simulations usually run more quickly than human-in-the-loop 
simulations, and can result in many more iterations being run. These many 
iterations can provide insightful answers to “what if’ questions not able to be 
answered in a slower-running human-in-the-loop simulation. 

The DHS’s Office of State and Local Government Coordination and 
Preparedness (SLGCP) recognizes the potential value of simulation in 
preparation for the TOPOFF program. Prior to TOPOFF 3, DHS reviewed 
100 models, simulations, and games (MS&G), evaluating their usefulness in 
preparation for and execution of the exercise. The result of this review was that 


‘Oaverill M. Law, and W. David Kelton, Simulation Modeling and Analysis, 3" Edition, 
2000, p. 1. 


some MS&Gs proved useful, but “no single group of products can be 
recommended that will support the requirements of all communities.”"' 

The Training and Doctrine Command Analysis Center-Monterey 
(TRAC-MTRY) is currently studying the use of simulation techniques to facilitate 
organizational learning. Figure 1 illustrates a methodology proposed by 
TRAC-MTRY to improve organizational learning, especially in support of the 
TOPOFF program. This iterative methodology includes scenario planning as the 
first step. An analyst models the scenario with a high resolution simulation, to be 
used as a training tool for exercise participants. Data that results from the 
simulation are analyzed and the base simulation adjusted as necessary to 
correct modeling deficiencies or perform “what if’ analyses. If decision makers 
are satisfied with the training achieved, the exercise can be executed and 


results documented. 


Planning 
Scenarios 





Model Scenarios Training Participants | 
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Adjust Scenarios of Simulation 





Execution of 


Initial Analysis TOPOFF Exercise 





Figure 1. TRAC-MTRY Proposed TOPOFF Methodology to Improve the 
Learning Process* 


"ThoughtLink, Inc. “Review of Models, Simulations, and Games for Domestic Exercises and 
Preparedness,” (2004). Retrieved on 5 December 2005 from the World Wide Web at 
http://www.ojp.usdoj.gov/odp/docs/Review_of MSG _SlimVersion.padf 


Figure 2 illustrates a revised methodology, adapted from the TRAC-MTRY 
organizational learning methodology. The following paragraphs are numbered in 
accordance with the steps in this methodology. 
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Figure 2. Adaptation of TRAC-MTRY's Learning Methodology ® 























1. Model Scenarios with Multi-Agent Simulation (MAS) 


MAS are low resolution models that are (relatively) easy to set up and 
alter, and are fast to run. Although it is not usually appropriate to use MAS to 
model an entire TOPOFF exercise, it is useful to model specific vignettes from 
the exercise plan. The power of MAS is enabled by the use of data farming to 


"Doris M. Turnage and LTC Jeffrey B. Schamburg, Facilitating Organizational Learning and 
Change Through the National Exercise Program, (December 2005). 


'Sibid. 


thoroughly explore the range of possibilities in these vignettes. The analyst 
leverages modern computing and supercomputing power to vary parameters and 


run the scenario thousands of times. 


2. Conduct Training or Analysis Runs 


MAS can be used for a variety of purposes. Within the context of this 
research, there are two major applications for low resolution simulation. First, the 
MAS may be used to train leaders and staffs. Inputting the scenario into the 
model and visualizing the events that unfold during a given model run can help to 
validate and streamline procedures. Second, data collection and the subsequent 
analysis can provide decision makers with insight into “what if’ questions that 


would be difficult to answer in another setting. 


3. Collect MAS Lessons Learned 


The data farming approach to analysis leads to a much more thorough 
exploration of the solution space than more complex, slower-running, 
high resolution models. By understanding the solution space’s “landscape,” we 
are able to consider “what if’ questions and gain unique insight into the 
problem.'* Analysts may use the insights gained from output analysis in two 
possible ways. The output may indicate a need to refine the MAS scenario, or 
rerun the experimental design to focus on a certain area. Alternatively, if the 
analyst is satisfied with the results of the data analysis, the insights are captured 
as lessons learned. The lessons learned from the low resolution simulation are 
used as a Starting point for a high resolution simulation, such as a 


wargame simulation. 


4. Develop Wargame Model Using MAS Lessons Learned 


MAS lessons learned can help provide insight into making necessary 


assumptions as the model is developed. During the execution of the 


“\TC Thomas M. Cioppa, “A Potential Role of Agent Based Models in Military Analysis,” 
Information Paper, (prepared December 2003). 


human-in-the-loop wargame, MAS lessons learned can assist decision makers in 
making effective, timely decisions. 


5. Conduct Wargame Incorporating MAS Lessons Learned 


The wargame can be either a timestep simulation or event-driven. 
Common to both approaches is that one side will act, the other will react, and the 
first side will then counteract, and then move to the next timestep or event. This 
type of simulation is high resolution, meaning that the output is more 
representative of actual events than the output from a low resolution simulation, 
and thereby more actionable." 

TRAC-White Sands Missile Range (WSMR) developed the Emergency 
Preparedness incident Command System (EPiCS), which is a computer-based, 
high resolution simulation. EPiCS is a version of the Joint Army Navy Uniform 
Simulation (JANUS), a well-known simulation used for many years to facilitate 
military wargaming. TRAC-WSMR developed EPiCS to facilitate wargaming in 
emergency response situations requiring multi-echelon and/or interagency 


communication and coordination." 


6. Satisfied With Results? 


The decision maker must review the products, insights, and 
lessons learned from the MAS and wargame. If satisfied that enough insight has 
been gained by the staff, the decision maker directs the staff to document the 
lessons learned and finalize the plan. The decision maker may require more 
information, different information, or a narrowly focused view of a small part of 
the response surface. In this case, the staff iterates the process of MAS 


modeling and wargaming. 


"Defense Modeling and Simulation Office Verification, Validation, and Accreditation 
Glossary. Retrieved on 20 March 2006 from the World Wide Web = at 
https://www.dmso.mil/public/library/projects/vva/glossary.pdf 


"EPICS homepage. Retrieved on 17 May 2006 from the World Wide Web at 
http://epics.astcorp.com 


7. Document Wargaming Lessons Learned 


Clearly and concisely documenting the lessons learned from the wargame 
is aS important as running a realistic simulation, and by doing so throughout the 
simulation process, we can disseminate the information to the people that need 
it. Wargaming lessons learned may be used in two ways: to refine the original 
MAS and then begin the process again or to finalize the exercise plan. 

Organizational learning does not stop with the final simulation; it continues 
throughout the TOPOFF exercise, and_ after. Agency leaders and 
first responders share the knowledge they gain during the execution of the 
exercise during “hot wash” after action reviews (AARs). The purpose of these 
short AARs is to capture important findings before too much time passes and 
memories fade. Following the TOPOFF exercise, a senior observer leads a 
comprehensive analysis of the exercise, to gain final insights into strengths and 
weaknesses of the response. Action facilitated by these insights is the beginning 


of preparation for the next TOPOFF exercise or next local crisis event. 


C. PROBLEM STATEMENT 


The SLGCP currently prepares for its biennial TOPOFF exercises without 
the use of computer simulation (e.g., MAS), and with limited use of 
human-in-the-loop simulation (e€.g., wargaming). Inclusion of simulation in the 
planning process of TOPOFF exercises could result in increased preparedness, 
more efficient resource allocation, and more meaningful lessons learned. The 
goal of this research is to provide a methodology for an analyst to use 
Pythagoras, or another MAS model, to simulate emergency response to a natural 
or man-made crisis event. The methodology developed in this thesis can be 
applied in many different situations, such as a future TOPOFF scenario in a 
major city, an annual force protection exercise at a military installation, or a small 


town’s response to a flood. 


D. SCOPE 


This research will focus on the police, fire, and medical response to a 
small terrorist cell detonating a bomb near the amphitheater in Baltimore’s 
Inner Harbor. This event replicates the event that occurred in exercise 
TOPOFF 3 in New London, Connecticut. TRAC-WSMR also replicated the 
New London event within their wargaming simulation, EPICS. TRAC-WSMR 
provided the specific scenario and force structure information used in the EPiCS 
run conducted in February 2006. This MAS development will replicate the first 
hour of TRAC-WSMR’s EPiCS run as closely as_ possible, using a 
multi-agent approach. 


This work is focused on the following research questions: 


e What is an appropriate methodology for use of a MAS environment 
in the modeling of emergency response to the simulated 
vehicle-borne improvised explosive device (VBIED) that explodes 
near the amphitheatre in Baltimore’s Inner Harbor? 


e What is the most appropriate mix of police, fire, and medical 
assets? 
e What is the most effective interagency communication architecture 


for emergency response to VBIED in Baltimore’s Inner Harbor? 
E. THESIS OVERVIEW 


The following chapters will provide specific background on the simulation 
area, details on how the model was constructed, how the data were collected 
and analyzed, and the resultant conclusions. Chapter Il presents detailed 
information to the reader that addresses the specific scenario analyzed for this 
research and the organizations included in the model. Chapter III addresses the 
development and application of a methodology for simulating emergency 
first response in a multi-agent environment. Chapter IV discusses the 
implementation of the model in detail, discussing the model’s settings, 
deficiencies, and limitations. Chapter V details the factors that are analyzed and 
the experimental design methodology. In Chapter VI, the author demonstrates 
the analysis of data gained from executing the experimental design. The final 


chapter presents an analysis of the data from the simulation, significant results, 
and possible follow-on work. 
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ll. BALTIMORE ATTACK SCENARIO OVERVIEW 


This chapter provides the reader with detailed information about the 
scenario and organizations studied as the basis for this research. The chapter's 
first section outlines the TOPOFF 3 exercise, the simulated bomb attack, and the 
reason for moving the location of the simulation to Baltimore from New London 
for this research. The chapter's second section provides information about the 


organizations simulated, including force structure and purpose. 


A. TOPOFF 3 SCENARIO 


The DHS conducted TOPOFF 3 during April 4-8, 2005. This exercise 
simulated two coordinated terrorist attacks. The first was a pneumonic plague 
attack in Union and Middlesex counties, New Jersey. The second was a 
chemical agent release (mustard gas) and high-yield explosive in the 
City of New London, Connecticut.'” 

The goal of this research is to develop a simulation methodology to model 
emergency first response in a crisis situation. The methodology must be robust, 
flexible, and capable of addressing various crisis scenarios in differing 


geographical settings. The author considered two ways to proceed: 


e Develop a detailed simulation and analysis of one vignette 


e Develop and analyze several smaller, less detailed vignettes 


To create a simulation that is realistic and widely applicable, it is 
necessary to be detailed in the approach of simulation design and analysis. The 
detailed approach enables the analyst to answer questions and solve problems 
that do not surface in a more cursory approach. It is detailed thinking and 
problem-solving that makes the simulation methodology robust. Therefore, the 


"’Department of Homeland Security, “A Review of the Top Officials 3 Exercise,” Office of the 
Inspector General, Office of Inspections and Special Reviews, p. 4. Retrieved on 
25 January 2006 from the World Wide Web at http:/Avww.dhs.gov/interweb/assetlibrary/OIG 
06-07 Nov05.pdf 
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scope of this research is limited to the high-yield explosive attack, which occurs 
during a festival that draws approximately 10,000 attendees. 

The explosion simulated during TOPOFF 3 occurred on April 4, 2005, 
during the annual Celebrate Connecticut festival held on the New London 
waterfront. A small terrorist cell, comprised of approximately five personnel, 
orchestrated a VBIED attack. The VBIED was the equivalent of 7,350 pounds of 
ammonium nitrate, approximately the same size as the bomb that destroyed the 
Murrah Federal Office Building in Oklahoma City." Figure 3 illustrates the 
affected area. 





Figure 3. Affected Area of TOPOFF 3 VBIED, New London, CT” 





'SE_mail from Dr. Julie Seton, EPiCS Program Manager, TRAC-WS\WR, titled “Message from 
Dan Edmonson — CT Incident Details for TRAC-Monterey Project,” 24 January 2006, 
office communication. 


'8Graphic retrieved on 17 May 2006, Using Google Earth; affected area provided by 
TOPOFF 3 Full Scale Exercise Final Planning Conference Brief, given March 2-3, 2005. 
Retrieved on 13 January 2006 from the World Wide Web at 
http://www.dhs.gov/interweb/assetlibrary/OIG_06-07 Nov05.pdf 
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B. MOVEMENT OF VIGNETTE TO BALTIMORE 


This simulation setting was moved to Baltimore due primarily to funding 
constraints. The high resolution model chosen for comparison is 
TRAC-WSMR’s EPiCS model. The terrain for Baltimore was available when the 
EPiCS simulation was built; however, the terrain for New London was not 
available and would have been both time-consuming and expensive to build. 

Movement of the event from its original venue does not decrease this 
research’s relevance. Certain necessary similarities exist between New London 
and Baltimore that make these cities an ideal pairing. Both cities celebrate 
festivals that draw approximately 10,000 people, resulting in similar response 
challenges due to crowd control. Both cities are located waterside, along an 
extensive pier network, providing the requirement for similar maritime response. 

There are advantages to simulating the event in a different setting than the 
original exercise. By setting the simulation in Baltimore, the results cannot be 
artificially “gamed” to look like the results of the exercise actually run during 
TOPOFF 3. In addition, transporting the scenario to different terrain shows that 
the methodology developed for simulation is applicable in different cities, not just 


in the place were the event was actually executed. 


C. DESCRIPTION OF FIRST RESPONDERS 


Three primary organizations provide emergency first response during a 
major civil crisis event: police departments, fire departments, and hospitals. The 
response from each organization occurs in stages, enabling these organizations 
to provide immediate support, not needing to wait until resources are assembled 
and staged. The following section is a synopsis of each organization’s force 
structure and purpose at each phase of the crisis response operation. 

The overall mission of the response forces, in priority order, is: 

e Control the crowd; localizing the bomb’s effect and minimizing 

further injuries 


° Identify and eliminate additional threats 
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e Identify the dead 


e Triage and stabilize the wounded 


1. Initial Response 


Initial response comes from the emergency services personnel that are in 
the immediate crisis area when the event occurs. The purpose of initial response 
is security and safety. The initial response force simulated in this exercise is 
comprised of police officers with two different missions. Police on foot patrol 
circulate throughout the Celebrate Baltimore area. Traffic police maintain 
positions at each intersection leading into and out of the festival area. 

Foot patrols in the Inner Harbor area serve several purposes. Their 
primary purpose is to provide police presence and security in the area. The 
secondary purposes are to conduct crowd control and provide medical 
assistance as needed. Traffic police from several separate police departments 
are stationed at the intersections in the Celebrate Baltimore area. Their primary 
purpose is crowd control, in addition to providing police presence in the area. 

There are 22 police officers who comprise the initial response police 
presence. Ten are on foot patrol (five patrols of two officers each) and one traffic 
officer stands at each of 12 intersections (see Figure 4 for a diagram of police 
locations). All initial response police carry similar equipment. They are armed 
with a standard issue 9mm semiautomatic pistol and wear Level III-A body 
armor, protecting them against bullet penetration from most handguns and some 
small caliber rifles. All police carry handheld radios capable of communicating 
throughout the simulation area.” 

Following the bomb attack, both patrol and traffic police maintain their pre- 
attack missions. The primary focus of the police in this situation is to maintain 
crowd control to limit the number of casualties caused by people running away 
from the attack site. While maintaining crowd control, police try to identify 
additional threats that may attempt secondary attacks. Police on foot patrol will 


20F mail from Dr. Julie Seton, EPiCS Program Manager, TRAC-WSMR titled “Force Files,” 
dated 20 January 2006, office communication. 
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continue to patrol, attempting to calm as many people as possible. Traffic police 
will maintain their positions at intersections, Keeping nonessential personnel out 
of the affected area in order to minimize congestion that will hamper 


follow-on forces. 


. 5 f a 
»*. a — ah ure A sia 


Traffic Police 


Foot Patrol 





Figure 4. Array of Initial Response Forces (Police)”’ 
2. Follow-On Response 


Follow-on response is comprised of police, fire, and medical personnel 
that are summoned to the scene of the attack by the central dispatch authority. 
The purpose of follow-on responders is to save lives, while police continue to 
provide crowd control and security. The fire department controls the threat due 
to fire (but will also assist in the establishment of triage points), takes the lead in 


hazardous material situations, and provides medical assistance with its organic 


"Graphic retrieved from Google Earth, 12 April 2004; locations are adapted from 
TRAC-WSMR EPiCS simulation conducted in February 2006. Free application downloaded on 
30 March 2006 from the World Wide Web at http://earth.google.com/download-earth.htm. 
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assets. Hospitals provide doctors, nurses, and support staff to triage, stabilize, 
and treat injured people on-site. In addition to site support, hospitals have the 
ability to provide surge support at their location. Figure 5 illustrates the initial 
location of follow-on responders and their distance from the location of the 
bomb blast. 
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Figure 5. Follow-on Response (Police, Fire, Medical)” 


Follow-on police response comes’ from the Central District 
Police Headquarters, approximately one-third of a mile from the scene of the 
explosion.” Police officers are diverted from their standard daily duties and 


phased into the response. Follow-on patrolmen have dual missions: control the 


“Background graphics and distances courtesy of Mapquest.com. Retrieved on 
30 March 2006 from the World Wide Web at http:/Awww.mapquest.com 


751 ocation of Central District Baltimore Police Headquarters. Retrieved on 30 March 2006 
from the World Wide Web at http://www.ci.baltimore.md.us/neighborhoods/facilities/polfire.html 
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crowd to minimize the effect of panic and identify and eliminate further terrorist 
threats. Providing basic first aid is a tertiary task. 

Follow-on police are equipped in the same manner as the _ initial 
responders. They are armed with a standard issue 9mm semiautomatic pistol 
and wear Level III-A body armor, protecting them against bullet penetration from 
most handguns and some small caliber rifles. All police carry handheld radios 
capable of communicating throughout the simulation area. 

In addition to the diverted patrols, the district's two Special Weapons and 
Tactics (SWAT) teams are located at the Central District Headquarters. The 
SWAT teams remain on call, staged and ready at headquarters, throughout the 
simulation. If an armed terrorist threat is identified, the threat location is radioed 
to the SWAT team. The SWAT team responds, eliminates the threat, and returns 
to the department headquarters to await another call. 

The SWAT teams have greater combat capability than the police patrols. 
SWAT team members carry an M4 carbine, in addition to the 9mm 
semiautomatic pistol. They also wear body armor that is analogous to the Army’s 
Interceptor body armor, which is more protective than the standard 
Level III-A body armor worn by patrolmen. SWAT team members also carry a 
handheld radio that can range over the entire simulation area. 

Medical responders include doctors, nurses, Emergency Medical 
Technicians (EMTs), and support staff from three area hospitals. Mercy Medical 
Center is located approximately nine-tenths of a mile north of the explosion. 
University Hospital is about one mile west of the scene. Saint Johns Hopkins 
Hospital is approximately two miles northeast. Hospitals respond to the 
emergency with teams of ten, comprised of doctors, nurses, support staff, and 
associated equipment. These teams arrive incrementally, every 45 minutes, 
after the event. The primary purpose of the medical teams is to triage the 
wounded, stabilize them, and evacuate them from the scene. In addition to going 


*“1 ocation of Baltimore hospitals. Retrieved on 30 March 2006 from the World Wide Web at 
http://www.local.com/results.aspx?keyword=hospital&location=Baltimore%2c+MD&radius=5 
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to the wounded and treating them in place, the medical teams have the 
responsibility of setting up triage points in the vicinity of the attack. 

Fire response in this simulation comes from two nearby fire stations. 
Station 23 is a large, fully capable station located approximately three-quarters of 
a mile west of the scene of the explosion. Station 33 is a smaller station that is 
about three-quarters of a mile south of the explosion.*® The primary mission of 
the fire department is to eliminate the threat of fire. If there is no threat of fire, the 
fire department will assist the police department with the evacuation of civilians 
from the area affected by the bomb blast. The fire department will also assist 
personnel from the area hospitals in setting up triage points. Stations 23 and 33 
respond to the scene with similar equipment: ladder trucks, pumping trucks, 
and medics. 

After the initial response and follow-on response by the first alarm 
responders, the incident commander makes a full assessment and will request 
second alarm responders. The mix of police, fire, and medical responders is 
dependent on the situation and location. This simulation does not address 
second alarm responders; simulating decisions made by the incident command 


post is beyond the scope of this research. 


D. DESCRIPTION OF CIVILIANS 


Ten thousand civilians are in the area of the Celebrate Baltimore festival 
when the terrorists detonate the bomb. There are three different classifications 
of civilians in the model: those unaffected by the bomb, those who are 
ambulatory wounded, and those that are stretcher wounded. There are four 
groups of the unaffected and ambulatory wounded civilians in the model: men, 
women, boys, and girls. Each group has separate characteristics in the model. 
All stretcher wounded (men, women, boys, and girls) have the same 
characteristics in the model. 


5! ocation of Baltimore fire stations. Retrieved on 30 March 2006 from the World Wide Web 
at http:/Awww.local.com/results.aspx?keyword=hospital&location=Baltimore%2c+MD&radius=5 
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1. Behavior of Unaffected and Ambulatory Wounded Civilians 


Unaffected civilians are aware of the bomb’s detonation; they want to 
leave the area. Civilians that are ambulatory wounded by the bomb behave in an 
unpredictable manner; that is, they panic. When civilians panic, they will respond 
to attempts by police, fire, and medical personnel to calm them. Panicking 
civilians will also respond to terrorist attempts to instill more panic. For example, 
if a civilian is moving toward a triage point because a policeman gave him 


directions, a terrorist can still deter the civilian and cause more panic. 


2. Stretcher Wounded Civilians 


Stretcher wounded civilians must be stabilized by hospital personnel 
(doctor, nurse, EMT) before they can move to a triage point. 


E. DESCRIPTION OF TERRORISTS 


Five terrorists orchestrate the simulated attack on the Celebrate Baltimore 
festival. Four terrorists have the purpose of increasing the level of panic 
experienced by the civilians in affected area. These four terrorists are in civilian 
clothes and attempt to blend into the crowd so as to not be readily identified by 
responding police officers. They do not have a weapon, but are armed with the 
knowledge of how to incite panic in a crowd. These terrorists focus on the areas 
affected by the explosion that have the largest civilian density. 

The fifth terrorist is a sniper that will directly attack responding agencies to 
disrupt their influence on the crowd. The sniper is armed with an AK-47 assault 
rifle and is wearing body armor. This sniper hides in a building during the initial 
response and first alarm response. After the first alarm responders have arrived 
on scene, the sniper will begin to shoot police, fire, and medical targets of 
opportunity. The goal of the sniper is to slow the emergency response and incite 
panic in the crowd by slowly attriting response forces. 

Now, with an understanding of the scenario and an overview of the forces 


simulated, a thorough discussion of the model follows. In addition, a detailed 
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description of methods used to replicate behaviors and characteristics of various 


weapon systems is included. 
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lll. MODELING METHODOLOGY 


This chapter provides a general background of simulation and MAS, and 
specific details of the methodology developed to model first response to a crisis 
situation in a MAS environment. It begins with a brief overview of simulation 
models and leads to a summary of multi-agent simulations and their place in 
combat modeling. The author establishes the reasons for using MAS to analyze 
the chosen situation and the selection of Pythagoras as the particular model 
used. This section concludes with a detailed, step-by-step discussion of a 
methodology that applies MAS techniques to modeling first response in a crisis 
situation, and a discussion of the methodology’s application. 


A. SIMULATION SUPPORT TO MILITARY ANALYSIS 


The Department of Defense (DoD) relies on simulation models to capture 
significant insights that enable senior leadership to make informed decisions. 
Insights gained from simulation models become especially important in areas in 
which experimentation is impractical or impossible. For example, it is nearly 
impossible to conduct actual physical experiments to determine the effectiveness 
of strategic or operational war plans, large-scale fielding of new force designs, or 
weapon system capability changes in actual conflict.” 

Traditional combat simulations are often complex, [high resolution] 

simulations that require a large amount [of] input data, an 

experienced and [well-trained] staff to operate and months to 
prepare specific scenarios for use. These issues, as well as slow 
runtimes, make traditional simulations both time and resource 
intensive. The time and resource requirements often force an 


analyst to conduct a very limited set of simulation runs thereby only 
focusing on a limited region of a [model's] input space.*’ 


8LTC Thomas M. Cioppa, “A Potential Role of Agent Based Models in Military Analysis,” 
Information Paper, (prepared December 2003). 


2"Ibid. 


23 


An example of why high resolution models have slow run times extends 
from simply capturing the effects of firing a round from a howitzer. 


To replicate a howitzer firing a projectile in a high resolution model, 
the analyst must know more information [than] just the classical 
‘trajectory in a vacuum’ physics problem. Instead, the analyst must 
take into account interior, exterior, and terminal ballistics. Each 
includes, but is not limited to, factors such as projectile square 
weight, propellant temperature, propellant moisture, muzzle velocity 
variation, and tube wear effecting interior ballistics, as well as 
meteorological atmospheric conditions such as air temperature, air 
moisture, wind direction, wind speed, and the rotation of the [earth] 
[affecting] exterior ballistics. These examples only name a few 
factors that the analyst could consider when modeling the howitzer 
firing the projectile. This process then repeats for every other 
howitzer in the battery, positioned at different locations, and any 
other munitions also fired. As such, a simulation requiring multiple 
munitions, from several platforms demands significant computing 
ability just to provide the decision maker with useful insights 
required for his decision.” 


The above is an example of the reason that simultaneous analysis of a 
wide range of issues is difficult with high resolution models. In addition, the 
traditional deterministic and even stochastic techniques that are used in these 
models cannot capture key elements of combat uncertainty. Factors such as 
synergy among actors, command and control influences, and human adaptivity— 
long considered strengths in the American military—are not well accounted for in 
traditional, high resolution models. The shortcomings of traditional models led to 
the search for a new tool.” 


An innovative class of low resolution simulation, known as MAS, emerged 
to complement and augment previously established, more computationally 
intensive, physics-based, simulation models. The role of MAS is not to replace 


8Charles A. Sulewski, “An Exploration of Unmanned Aerial Vehicles in the Army’s Future 
Combat Systems Family of Systems,” Masters Thesis, Naval Postgraduate School, Monterey, 
CA, December 2005. 


*°Raymond R. Hill, et al., “Some Experiments With Agent-Based Combat Models,” Military 
Operations Research, Vol. 8, No. 3, 2003, pp. 17-28. 
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high resolution models, however, MAS increasingly proves useful to the DoD in 
two primary areas. 


MAS may be used in exploratory analysis, to gain quick insight and 
narrow the focus of many factors, parameters, and variables to expedite using 
high resolution, physics-based simulations.*° Narrowing the field of factors to 
explore saves time and money at the beginning of a simulation study. MAS is 
also used to address key objectives that would otherwise require extensive use 
of resource-intensive, physics-based models. The analyst may switch back and 
forth between two models to gain advanced scenario insight.*' 

Analysis of a MAS leads to a relatively fast, thorough exploration of the 
solution space associated with the given problem. MAS offers quick scenario 
generation, fast run times, rapid data turn around, and permits the analyst to 
consider many alternatives in a short amount of time. MAS complements and 
augments physics-based models, permitting analysts to examine the problem 
over a greater range of plausible possibilities. 

MAS is an appropriate method to help an analyst provide insights that 
support decision makers in situations involving crisis response. Crisis response 
involves human decision-making, which can be unpredictable, especially under 
stressful situations. Decisions made by individuals can lead to global patterns or 
emergent behavior. MAS can help identify possible patterns and behaviors. 
MAS also offers the analyst the ability to run a model thousands of times to gain 
insight into hundreds or thousands of different “thought processes.” In many 
circumstances, especially at the beginning of analysis, this insight is more 
valuable than providing a handful of point estimates that come from relatively 


long, expensive, high resolution runs. 


S°LTC Thomas M. Cioppa, “A Potential Role of Agent Based Models in Military Analysis,” 
Information Paper, (prepared December 2003). 


Ibid. 
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B. PYTHGORAS BACKGROUND 


The author chose Northrop Grumman’s Pythagoras as the agent-based 
simulation modeling tool to support this research because its developers included 
several features in this model that are especially useful in modeling crisis 
response. Crisis response is inherently complex: a multisided event with many 
interrelationships and a requirement to gradually escalate force. Pythagoras 


offers a set of capabilities that is unique in the area of agent-based simulations:* 


° Incorporates soft rules to distinguish unique agents 
e Uses desires to motivate agents into moving and shooting 
e Includes the concept of affiliation (established by sidedness, or 


color value) to differentiate agents into members of a unit, friendly 
agents, neutrals, or enemies 


° Allows for behavior-changing events and actions (called triggers) 
that may be invoked in response to simulation activities 


° Retains traditional weapons, sensors, and terrain 


Northrop Grumman _ initially developed Pythagoras in support of 
Project Albert, a U.S. Marine Corps-sponsored international initiative focusing on 
human factors in combat and noncombat situations. They have updated the 
model as needed and as requested, including several updates during the 


development of this thesis. 


The developers of Pythagoras built this model with the intent that an 
individual without a hard science degree could learn to effectively use the model 
within eight hours.* The company’s goal of simplicity paid off. Pythagoras offers 
a simple-to-use Graphical User Interface (GUI), including drop-down window 
capabilities much like many Windows-based applications. Within a few short 
hours at the console, a Pythagoras novice can input a meaningful scenario and 
start gaining insights into the problem at hand. 


*?Henscheid, Z., Pythagoras User’s Manual, Version 1.9, February 2006, p. 18. Retrieved 
from the World Wide Web on 5 February 2006 at http://www. projectalbert.org/downloads.html. 


BS Ibid. 
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C. METHODOLOGY OVERVIEW 


The main objective of this research is to provide a methodology by which 
emergency first response to a crisis event can be simulated in a MAS 
environment. This methodology expands on areas two, three, and four of 
Figure 2. Figure 6 provides an illustration of the developed methodology, 
followed by a description of the diagram. Many of the steps are interrelated, 
iterative, and flexible; the given situation may require some of the steps to be 
accomplished in a different order than outlined in Figure 6. 
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Figure 6. Methodology for Simulating Emergency First Response to a Crisis Event 
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The focus of this section is on steps 1 through 7 of the methodology. The 
author explains steps 8 and 9 in detail in Chapter IV (Model Implementation). 
Chapter V (Design Methodology) focuses on steps 10 and 11. Chapters VI 
(Data Analysis) and _ VII (Conclusions and Recommendations _ for 
Future Study) detail the procedures for steps 12 and 13, respectively. 


The vignette modeled for this thesis is a terrorist attack in a large city 
(Baltimore) during a festival. This simulation is an application of the 
methodology, but this methodology is applicable to response in many other 


crises: manmade and natural, large scale and small. 


ie Identify Event Type and Location 


The first step in addressing a problem in a scientific manner is to define 
the problem. Simulating emergency first response begins with knowing the event 
and the location. Response to a terrorist attack is different than response to a 
natural disaster. Though the resources and priorities of work differ between a 
natural and a man-made attack, the methodology used for simulating them is 


the same. 


2. Identify Level of Governmental Response 


The analyst must consider several factors when deciding on the 
appropriate level of governmental response to analyze. Identifying the 
requirements in SOPs that govern crisis response in the appropriate situation is 
the first step, since procedures in different crisis events call for different levels of 
response. For example, federal agencies will respond to a terrorist attack without 
the requirement of being requested by the local government. However, in a 
natural disaster, the local government may need to request federal assistance. 


In addition to verifying the procedurally appropriate level of response, the 
analyst must consider the goals of the decision maker. In a situation that would 
usually involve the response of the federal government, the decision makers 
could be city planners who are seeking to validate local plans without 
federal assistance. 
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3. Identify Level of First Response 


Identifying which response agencies to include is another important step in 
scoping the problem. Again, the modeler must identify what the decision maker 
deems is important. If the simulation is meant to support a police chief's decision 
concerning force allocation, inclusion of other agencies may not be a 
requirement. In a simulation meant to support federal emergency response, it is 


important to simulate all facets of emergency response. 


4. Establish Interagency Communication 


If all agencies being simulated belong to the same organization, lines of 
communication for information sharing are most likely in place. Some 
organizations may have different parents; the next step in this simulation 
methodology is to establish communication between like offices in the different 
organizations. Information sharing is the key to a realistic, credible simulation of 


crisis response. 


5. Enumerate Focus Areas and Questions to be Answered 


With the background for the simulation set and the lines of communication 
open, the focus shifts to the goals the decision makers are seeking to achieve. 
The analyst must identify areas of interest in which the decision maker wants to 
gain insights. Narrowly scoped research questions will guide the direction of the 
simulation later in development. With the areas of interest and research 
questions developed, the analyst can review and prioritize agency SOPs for 


priorities of work needed to address the focus areas and research questions. 


6. Identify Priorities of Work Relevant to Focus Areas 


Identify priorities of work that occur automatically after the crisis event; 
also, identify priorities of work that require decisions to be made after the event 
occurs. The simulation can easily include decisions that are made before the 
event. However, MAS are simulations that are “human-out-of-the-loop.” Actual 
human decisions stop when a constructive MAS starts. When the modeler 
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encounters a place in the simulation where an actual human decision should be 
made, i.e., a decision beyond the scope of the simulation’s agents, there are two 
ways to proceed. First, the modeler can end the simulation at the point of the 
human decision. Alternatively, if the decision maker needs the simulation to run 
longer, the modeler can make the decision before the model is run. Each 
decision is fully documented, complete with the reasons for making the decision. 


During the process of establishing priorities of work, the analyst begins to 
develop the list of factors included in the model that are most important to the 
decision maker. During the experimental design stage of this methodology, the 
analyst will vary these factors to conduct the analysis that will answer the 
questions posed at the beginning of the project. 


7. Choose the Model 


When priorities of work are established and prioritized, the analyst must 
recognize the actions taken to accomplish those priorities. The simulation must 
reflect reality. The actions taken to accomplish priorities of work are important 
because it is these actions that determine which simulation model to use. 


It may be better to use Pythagoras to simulate a model in which the 
decision maker is interested in leadership qualities, behavior tolerances, or 
affiliation changes. The decision to use Pythagoras to model the TOPOFF 
scenario is discussed in Section III.B. Alternatively, it may be most appropriate 
to use Map Aware Non-uniform Automata (MANA) if the decision maker is 
interested in communication architecture or situational awareness.** The 
decision maker may choose NetLogo if interested in the dynamics of crowd 


control or mob effects.*° 


“David P. Galligan, Mark A. Anderson, and Michael K. Lauren, Map Aware Non-uniform 
Automata, Version 3.0, July 2004. 


*Uri Wilensky, NetLogo User’s Manual, Version 3.1, April 2006. Retrieved from the 
World Wide Web on 19 May 2006 at http://ccl.sesp.northwestern.edu/netlogo/docs 
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8. Create Small Vignettes 


After choosing the model, begin modeling small portions of the overall 
scenario. Start by modeling and debugging similar agencies or priorities, or work 
separately to simplify interactions among agents. See Chapter IV 
(Model Implementation) for detailed information on how to create vignettes in a 
MAS, specifically, Pythagoras. 


9. Merge Separate Agency Vignettes and Debug 


After debugging separate vignettes, merge the separate vignettes into one 
large scenario. Add and debug one vignette at a time to minimize time spent in 


locating errors in the model. 


10. Calibrate the Model 


Model calibration gains the developed simulation a measure of credibility. 
The model developer compares the outputs of the simulation to other, previously 
established outputs. For example, an analyst may compare the outputs of the 
model to events that actually happened, if applicable. Another way to calibrate a 
model is to statistically compare the model’s outputs to that of another model, 


possibly a validated or accredited model. 


11. List Factors of Interest 


Developing factors of interest is an ongoing process that begins when the 
decision maker’s priorities are established. Most relevant work on this step 
occurs during the time that the scenario developer is creating, debugging, and 
merging vignettes. After the vignettes are merged into one scenario and 
debugged, the factor list can be formalized, complete with the corresponding 
levels that provide the decision maker with desired insights. The analyst must 
clearly identify the desired effects of varying a given parameter. Chapter V 
(Design Methodology) provides detailed information on how to accomplish steps 
10 and 11. 
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12. Varying Factors of Interest in Experimental Design 


After the simulation is complete and debugged, use the power of modern 
computers and supercomputers to “data farm.” 
Data farming is the process of using a 
[high-performance] computer or computing grid to run 
a simulation thousands or millions of times across a 
large parameter and value space. The result of 
Data Farming is a ‘landscape’ of output that can be 


analyzed for trends, anomalies, and insights in 
multiple parameter dimensions.*° 


13. Analyze Data 


It is through the analysis of the data “grown” by the use of an efficient 
experimental design and data farming that insights and lessons learned are 
obtained and documented, and then provided to the decision maker. See 
Chapter VI (Data Analysis) for information about applicable data analysis 


techniques and results from the Baltimore Attack scenario. 


14. Document Insights and Lessons Learned 


Following the analysis of the model and factors, it is possible to refine the 
model and repeat the experiment. If the model, experimental design, and 
resultant data analysis meet the decision maker’s needs, the analyst must 
document findings and lessons learned. See Chapter VII (Conclusions and 
Recommendations for Future Study) for documentation of the findings and 


recommendations resultant from analyzing the Baltimore Attack scenario. 


D. METHODOLOGY APPLICATION 


This section demonstrates how the previously described methodology 
applies to simulating the TOPOFF 3 scenario. Each step is enumerated, 
followed by a concise bullet comment that summarizes how the step is achieved 
in this research. 


*°Definition of Data Farming. Retrieved on 9 May 2006 from the World Wide Web at 
http://en.wikipedia.org/wiki/Data_farming 
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Identify Event Type and Location 


e TOPOFF 3 scenario, large bomb VBIED, New London, CT 


e Move scenario to Baltimore with same _ characteristics 
as above 


Identify Level of Governmental Response 


° Focus on city planning, do not use federal and/or 
state assets 


Identify Level of First Response 


e Police, fire, and medical response 


Establish Interagency Communication 


° Not applicable in this simulation, proof of concept exercise 


Enumerate Focus Areas and Questions to be Answered 


e Communication: What is the most effective interagency 
communication architecture for emergency response to 
VBIED in Baltimore’s Inner Harbor? 


e Determination of Assets Required: What is the most 
appropriate mix of police, fire, and medical assets? 


Identify Priorities of Work Relevant to Focus Areas 


e Identify/neutralize additional threats 

e Coordinate initial and follow-on response 
e Identify dead and wounded 

° Stabilize wounded, move to triage site 

° Restore calm 


Choose the Model 


e Use Pythagoras 


Create Small Vignettes 


e Terrorist agitators incite panic in crowd 
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10. 


11. 


12. 


13. 


e Police on patrol, looking for wounded, threats 


° Arrival of medics, treatment of stretcher wounded 

e Incident command post, fusion of communication across 
different agencies 

e Integration of SWAT team, response to threat 

e Arrival and deployment of fire department 

° Arrival and deployment of police follow-on response 

e Terrorist gunman hides and attacks targets of opportunity 


Merge Separate Agency Vignettes and Debug 


e The small vignettes are merged into one overall simulation, 
see Chapter IV (Model Implementation) 


Calibrate the Model 


° A rigorous way to calibrate this simulation model is to 
generate a distribution of outputs, then compare the results 
to one EPiCS run, or a distribution of outputs gained from 
several EPiCS runs. The capability does not currently exist 
to run several EPiCS simulations and capture the 
needed data. 


e The author calibrated this model through the use of modeling 
subject matter expertise at the Naval Postgraduate School 
and his own personal expertise in the area of emergency 
first response operations. This calibration was conducted 
using a qualitative comparison of the outputs this simulation 
and EPiCS can provide. 

List Factors of Interest 


e See Chapter V (Design Methodology) 


Varying Factors of Interest in Experimental Design 


e See Chapter V (Design Methodology) 


Analyze Data 


e See Chapter VI (Data Analysis) 
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14. Document Insights and Lessons Learned 


e See Chapter VII (Conclusions and Recommendations for 
Future Study) 


This section has provided the reader with a general overview of 
simulations and specific information about the MAS, Pythagoras. The author 
establishes and explains a methodology by which an analyst can use MAS to 
model emergency first response to a crisis event. The section concludes with an 
overview of how Pythagoras can be used in the application of the methodology to 
model a TOPOFF scenario. In subsequent sections, the “nuts and bolts” of the 
model are explained, including settings, assumptions, and limitations. In 
addition, the process of experimental design in this simulation is discussed, with 


the resultant data analysis and conclusions. 
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IV. MODEL IMPLEMENTATION 


This chapter provides a detailed, step-by-step discussion of the 
techniques used to model emergency response to a crisis in a MAS. The 
modeling described here corresponds to steps 8 and 9 of the methodology 
outlined in Figure 6 and subsequently discussed. Additionally, this chapter will 
touch on step 10, calibration of the model. The following includes details that 
address Pythagoras’s model settings specifically, but can be generalized to be 
applicable to other models. The chapter concludes with a discussion of Known 


“bugs” in the model and the model’s limitations. 


A. MODELING TECHNIQUE 


This section contains general information that can be used to create the 
scenario within most any MAS. In addition, this section contains detailed 
information that describes how this simulation was created in Pythagoras. 
Appendix A (Model Implementation) contains Excel tables and Pythagoras 
screen shots that illustrate ideas discussed here in writing. The first section of 
Appendix A outlines the creation of the environment in which the simulation 
occurs. It is the translation of the actual world into the simulation world. In the 
subsequent sections, the author describes the different types of agents in the 
scenario and the interactions among agents in each major event. The detailed 
discussion includes the goals of each organization involved, modeling techniques 
used to capture the effects of organizational operations, and _ the 


assumptions made. 


The author does not claim that the tactics modeled in this simulation are 
optimal emergency response tactics, techniques, and procedures (TTPs). This 
simulation does provide a proof of concept that it is possible to model emergency 
response priorities and TTPs in a multi-agent environment. The power of this 
tool is that it is possible to quickly adapt this model and this scenario to analyze 
many different force structures, TTPs, organizational priorities, and even other 


crisis events. 
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1. Overall Model Configuration 
a. Scaling the Model 


The first step of using a simulation model is scaling the model’s 
workspace to represent the area of interest. Traditional units of measure include 
feet, miles, meters, or kilometers. This scenario includes a “terrain box” that is 
four million square feet of the Baltimore Inner Harbor, 2,000 feet long by 
2,000 feet wide (see Figure 7). The unit of measure in Pythagoras is the pixel; 
the size of the model’s terrain is 1,000 pixels by 1,000 pixels. In this vignette, 
one pixel represents two feet. Appendix A includes a detailed discussion about 





Figure 7. Comparison of Actual Terrain®’ in Baltimore’s Inner Harbor to Terrain in Model 


°’Graphic retrieved from Google Earth, 12 April 2004; locations are adapted from 
TRAC-WSMR EPICS simulation conducted in February 2006. Free application downloaded on 
30 March 2006 from the World Wide Web at http://earth.google.com/download-earth.htm. 
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b. Timestep 


Time in Pythagoras is measured by timesteps. It is important to 
give careful consideration to the determination of the translation from seconds to 
timesteps. Agents move from their previous position to their subsequent position 
by jumping directly to the desired position. Agents do not pass through the pixels 
between positions. A timestep that represents a long period of real-time could 
result in agents “jumping” past each other and not interacting. A timestep that 
represents too little real time could result in the simulation requiring a vast 
amount of CPU time to run. For this vignette, the author chose one timestep to 
represent four seconds, resulting in a simulation of one hour of time that runs for 


900 timesteps. 


Cc. Terrain 


Terrain is defined by its characteristics. A given terrain feature will 
have a certain height; it may offer visibility through it, restrict movement, and 
have a protective value. Different types of terrain, as seen in Figure 6, have 
different colors, which are chosen by the user, not the developers. Gray terrain 
in one simulation may be easily trafficable concrete; in another simulation, gray 
could be an impenetrable wall. In Pythagoras, concealment occurs in three 
“bands’—A, B, and C. These bands are associated with the possible selections 
for available sensors in the model. More discussion about sensing bands follows 
in Section IV.A.1.f. See Table 1 for Pythagoras terms associated with its terrain. 
Specific characteristics for all weapons included in this simulation are located in 
Appendix A, Section A.1.c. 

[see [sana | ie [So 
Terminology Language 


Height (feet Height (pixels 0 — 1,000 


Sight Visibility 0.0 — 1.0 
Trafficability Movement Factor 0.0 — 1.0 
0.0= 1.0 





Table 1. Translation of Pythagoras Terrain Terms 
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Concealment, movement factor, and protection are multipliers that 
affect the probability of detection, rate of movement, and probability of being 
killed. For example, a wall made of cinder blocks may have a height of 20 feet 
(10 pixels). A person cannot see through the wall, so the concealment factor is 
1.0. The probability of detection results from the following formula: 


Actual P, =[1-(Concealment Factor)] * Baseline P,, 


A person cannot walk through a wall that is 20 feet tall, thus the 
movement factor would be 0.0; therefore, the movement rate is 0: 0.0 * standard 
movement rate. 

Suppose the weapons included in the simulation cannot shoot 
through a cinder block wall. The protection value would be 1.0. The probability 
of kill for a weapon trying to shoot through the wall is 0, because Pythagoras 
includes (1-protection factor) as a multiplier in determining actual probability of 
kill for a given situation. The author addresses movement rates in greater detail 


in Section IV.A.3.a. See Table 2 for the types of terrain included in 


this simulation. 


Movement 
Color Concealment Protection 


meg coroete {| 0.0 {0.0 | 4.0 | __0.0__ 
Road | 0.0 


Tan inside Bldg [0.0 [00 | 
| | Rubbling Spies [0s 02 00 
Red [Triage Site | 0.0 | 00 | 10 | 00 


Table 2. Terrain Colors Used in Baltimore Attack 





d. Weapons 


One way that an agent interacts with other agents is through the 
use of “weapons.” Weapons in most simulation models, and Pythagoras in 
particular, are not limited to killing enemies. Table 3 describes the different types 


of influence that Pythagoras weapons can impart. In general, weapons influence 
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the target’s health, sidedness, or another user-selected attribute. Changes in 


sidedness and attributes are discussed in detail later in this section. 


Weapon Effect 
Killing Decrease Target’s Health Policeman’s Pistol 


Increase Target’s Health EMT’s Medical Kit 
Paintball Incremental Side or Attribute Change | Propaganda Leaflets 


Table 3. Pythagoras’s Weapon Effects 





Table 4 provides a breakdown of the different weapons used in this 
simulation; who uses them and their effects. Appendix A includes detailed 
weapon information. The United States Army Materiel Systems Analysis Activity 
(AMSAA) provided weapon characteristics for the M4 rifle, M9 pistol, and AK-47 
assault rifle. Characteristics of other weapons are the result of the author's 
experience as a military policeman, in addition to informal conversations with 
police, Special Forces operators, and other subject matter experts. More 
detailed information about agent side properties is included in Appendix A, 
Section A.1.d. 


WPN : 
[wowen [SSS | wenoncorer [Toom | eet 


[Orders |_—P__| Police, EMTs, Firemen | Friends_| Calm Panic, Send to Med Station | 


*K denotes “killing” weapon, lethal effects 
**P denotes “paintball” weapon, nonlethal effects 
**R denotes “restorative” weapon, healing effects 





Table 4. Weapons Modeled in Baltimore Scenario 
The effect that a given weapon has on its target is calculated by 


multiplying the following factors: 


e Probability of hit associated with the weapon, as a function of range 
e Shooting agent’s marksmanship 


° Probability of kill of the weapon (given a hit) 
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e The effectiveness of the weapon (lethality) 
e Vulnerability of the agent being shot 


° Protection factor offered by the terrain 


e. Agent Side Property 


Side property is a feature that is unique to the Pythagoras model, 
allowing the inclusion of many different sides in a simulation, not just friends and 
enemies. The modeler assigns sides to the agents using color combinations of 
blue, green, and red. Each color is established on an integer scale between 0 
and 255. The modeler then establishes a color radius to determine unit 
membership, friendship, neutrality, and who is an enemy. Agents will assess 
each other’s color and compare it to their own color to determine affiliation, 
based on the distance between the two agents’ colors and the unit, friend, and 


enemy color radii. 


Table 5 shows a simple example of friend, neutral, and enemy 
affiliation, using only the color blue. 


Friend Enemy 
Radius Radius 


255, 0, 0 255, 0, 0 


255, 0, 0 
255, 0, 0 
255, 0, 0 150, 0,0 





Table 5. Determination of Affiliation by Color 


Pythagoras enables the modeler to use any combination of blue, 
green, and red to determine affiliation. In using combinations of two or more 
colors to determine affiliation, the modeler must decide on one of two methods to 
calculate distance between the agents’ colors. 


Manhattan Distance (used in this thesis) is the result of adding the 
color differences in each of the three colors, and is calculated by using the 


following formula. 


d =|Blue, —Blue,|+|Green, -Green,|+ |Red, —Red,| 
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The mathematical, or straight line, distance between the two colors, 
also called the Euclidean Distance, is calculated using the following formula. 





d= /lue, —Blue,)’ + (Green, —Green,)* + (Red, —Red,)’ 


There are many sides represented in this simulation. There are the 
public servants (police, fire, and medical personnel), the public (civilians), and the 
danger to the public (terrorists). Police are separated into several different sides, 
representing police from different districts. The terrorists are also divided into 
two sides: those that incite panic and those that attack the first responders. 
Table 6 highlights the different sides and their relationship to each other. 


Unit Friend {Enemy | 
_Inner Harbor Police | 1 | 6 TB 
2 Traffic Police 2 | 16 {76 _| 


4 
Hospi 





Table 6. Relationship Between Sides 
f. Sensors 


In Pythagoras, a sensor’s ability to detect an agent is based on 
three factors: the detection ability of the sensor, the terrain’s concealment factor, 
and the agent’s detectability. Each of these factors occurs over three bands: 
A, B, and C. These three bands enable the modeler to simulate three different 
types of detection, e.g., visual spectrum, thermal detection, and infrared. A piece 
of terrain may deny the agent an ability to see with the naked eye, but visibility is 
possible with an infrared or thermal device. 

In this simulation, agents have one sensor, eyes, which detect in 
the visual spectrum. All agents are completely detectable. The limiting factor is 
the concealment factor of the terrain, previously outlined in Table 2. Figure 8 


43 


shows the effectiveness of eyes by range, across terrain with no concealment 
factor. To find the actual probability of detection, use the following formula: 


Actual P, =[1—(Concealment Factor)] * Baseline P, 





Probability of Detection for Eyes 














Probability of Detection 
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Figure 8. Baseline Probability of Detecting a Target Using Eyes 
g- Communication Devices 


Communication devices are available for agents to pass 
information to each other. In general, there are two major types of 
communication devices: those that require line of sight and those that have 
broadcast capability and therefore do not require line of sight. Pythagoras 
enables modelers to vary the probability of successful communication with a 
given device. By changing the probability of successful communication, the 
analyst can make an assessment concerning the importance of a solid 
communication network in a given situation. 

In this simulation, each agent is given a voice and a line of sight 
communication device. See Figure 9 for the probability of successful 
communication for an agent using its voice. Each police, fire, and medical unit 


has a broadcasting communication device that simulates the handheld radio 
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commonly carried by each agency in their daily duties. The handheld 


communication device is capable of communicating over the entire simulation 
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Figure 9. Probability of Successful Communications Using Voice 
h. Agent “Personalities” 


A definition for personality is “the total of qualities and traits, as of 
character and behavior, that are peculiar to a specific person.”” 

Agent personalities in this MAS result from the effects of their 
behaviors on the surrounding environment, including other agents. Specifically, 
the types of behaviors that are used to define agent personalities are: the desire 
to hold fire in the presence of enemies (aggression), leadership and obedience, 
movement desires, and alternate behaviors that are triggered by a certain action 
or condition. 

(1) Hold Fire Desire. In Pythagoras, the modeler controls an 
agent’s hold fire desire by assigning the agent a number on a scale of Oto 1. A 


Definition of Personality. Retrieved on 10 May 2006 from the World Wide Web at 
http:/Awww.dictionary.com 
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hold fire desire of 1 is analogous to restrictive rules of engagement (ROE), in 
which the agent will not fire unless fired on. A hold fire desire of 0.5 means that 
half the time when an agent sees an enemy, that agent will fire first, and half the 
time that agent will not fire unless fired on. A hold fire desire of 0 is analogous to 
a “free-fire zone” in which the agent will shoot as soon as an enemy is identified. 
When the decision to engage has been made, the agent must decide from one of 
possibly three weapons. The modeler must identify if the agent will select the 
weapon with the highest probability of kill, the lowest, or the medium Px. 

(2) Leadership and Obedience. Pythagoras enables the 
creator of the simulation to incorporate leadership and obedience into the agents’ 
personality profiles. There are two different types of leaders in Pythagoras: 
charismatic and hierarchical. When exposed to charismatic leadership, the 
agents within the unit will respond to the leader with the highest leadership value, 
a value between 0 and 100. In hierarchical leadership, the agent will follow the 
leader that has the smallest leadership value higher that its own leadership 
value. Agents follow their leadership in accordance with their obedience value, 
established by the simulation’s creator on a scale of 0 to 1. An agent with an 
obedience value of 0.5 will only follow orders half of the time. 

Leadership and obedience each have a “tolerance” 
associated with them. This tolerance provides a way for the modeler to capture 
the effect of people interpreting situations differently. The tolerance is on the 
same scale as the initial trait. A small tolerance would indicate a predictable, 
disciplined outcome. A large tolerance indicates more randomness, with possibly 
less discipline and less predictability. 

(3) Movement Desires and Alternate Behaviors. The 
developers of Pythagoras provide the modeler with several distinct movement 
desires that help to shape each agent’s simulated personality. Figure 9 provides 
a depiction of available movement desires. Each movement has a 
corresponding desire and desire tolerance. The modeler decides which desires 
to activate within each agent. These desires are adjusted by the modeler until 
the agents move as desired. Every desire is assigned a number between 
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0 and 100. An agent with a high movement desire is more likely to execute that 
behavior. Event movement desire has a tolerance associated with it, similar to 
leadership and obedience. 

In addition to the desire fields, there are movement desires 
that correspond to a desire to move away from or toward other agents. These 
desires are based on counts or ratios. For example, an agent may have the 
desire to move away from an enemy if the force ratio is greater than a certain 
number. Similar desires correspond to distances. An agent may have the desire 
to move toward the nearest unit member if that unit member is farther away than 
a certain threshold. The movement desire tab enables the modeler to explicitly 
define these thresholds. As with the desired tolerance, the modeler can implicitly 
capture a measure of discipline or organization by using the tolerances 
associated with ratios or distances. 

The agent “decides” which desire to select by virtue of a 
choice the modeler makes, illustrated in Figures 10 and 11. The agent can move 
in accordance with its highest desire. It can also select a desire randomly, based 
on the draw of a random number. In addition, the decision may be made by 
calculating a weighted, vector average of the active desires. Finally, the modeler 
may decide to enable the agent to move in accordance with a weighted average 
of its top two desires. 
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Movement Method: [Highest Desire | 


Title 
Toward The Leader If Farther Than 


Away From The Leader If Closer Than 

Toward Closest Unit Member If Farther Than Highest Desire 

Away From Closest Unit Member If Claser Than 

Toward Furthest Unit Member If Farther Than | 

Toward Closest Friend If Farther Than i F 
Away From Closest Friend If Claser Than jRandom Desire 
Toward Furthest Friend If Farther Than Average Desire 
Toward Injured Friend 0.0 0.0 
Toward Friend Needing Fuel 
Toward Friend Needing Resourcex 
Toward Friend Needing ResourceY 
Toward Friend Needing ResourcezZ 
Toward Friend Giving Fuel . 
Toward Friend Giving Resourcex Possible methods 
Toward Friend Giving Resource to decide which 
Toward Friend Giving ResourceZ . . 

Toward Nearest Enemy If Farther Than desire to pick 
Away From Nearest Enemy If Closer Than 










CountiForce Ratio Tolerance 







































































Toward Nearest Enemy If Fewer Than Count 0 0 
Away From Nearest Enemy If More Than Count 0 0 
Toward Nearest Enemy ifForce Ratio More Than —_‘|0.0 0.0 
Away From Nearest Enemy If Force Ratio Less Than|0.0 0.0 





Toward Next Waypoint 
Toward Final Objective 
Maintain Last Course 
Select Random Direction 
Stay in place 


Movement Method: [Highest Desire | ~_ 


Title Desire Desire Tolerance Distance Distance Tolerance 
Toward The Leader If Farther Than 
Away From The Leader If Closer Than 
Toward Closest Unit Member If Farther Than 
Away From Closest Unit Member If Closer Than 
Toward Furthest Unit Member If Farther Than 
Toward Closest Friend If Farther Than 
Away From Closest Friend If Claser Than 
Toward Furthest Friend If Farther Than 
Toward Injured Friend 
Toward Friend Needing Fuel 
Toward Friend Needing Resourcex 
Toward Friend Needing ResourceY 
Toward Friend Needing ResourcezZ 
Toward Friend Giving Fuel 
Toward Friend Giving Resourcex 
Toward Friend Giving ResourceY 
Toward Friend Giving Resourcez 
Toward Nearest Enemy If Farther Than 
Away From Nearest Enemy If Claser Than 
Toward Nearest Enemy If Fewer Than Count 
Away From Nearest Enemy If More Than Count 
Toward Nearest Enemy if Force Ratio More Than 
Away From Nearest Enemy lf Force Ratio Less Tha 
Toward Next Waypoint 
Toward Final Objective 





































































































This agent has a desire 
Maintain Last Course foo N __ {0 of 100 to move ina 








Select Random Direction 


Stay in place random direction 




















Figure 10. Pythagoras Movement Desires 


Behaviors define agents’ personalities in a MAS. As the 


simulation progresses, the agents respond to stimuli in their environment in 
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different ways. These alternate behaviors define the agents’ personalities as 
much as the behaviors activated at the beginning of the simulation. The author 
of the simulation creates alternate behaviors to capture an effect that occurs 
within the simulation. For example, at timestep one, when the bomb explodes in 
this simulation, it triggers the civilians to panic. The specific trigger is the 
timestep; at timestep one, the properties of the civilians change to reflect feeling 
the effects of the bomb, then panicking. There are several event triggers 


provided in Pythagoras, some examples of which are illustrated in Table 7. 


Triaaer hoe: Alternate Behavior Alternate Behavior 
99 Value Triggered Ordered 


[Absolute Timestep | 60 | EMTstraveltosite = |N/A 


N/A = Not Applicable. 


Table 7. Example Alternate Behavior Triggers 





The author defined the panic behavior by changing certain 
qualities of the agents affected, including rate of movement, movement desires, 
and a establishing a need to be “calmed.” See Table 8 for a list of alternate 
behaviors developed for this thesis. Behaviors of each individual class of agent 
in the simulation will be discussed later in this chapter, in each respective 
section, one for each class. 
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Alternate Behavior Agent Class : eae 


Provide aid, bring SW civs to triage point 





Table 8. List of Alternate Behaviors 


Now that the overall setting for the simulation model is 
established, the following section will provide information about the 
representation of the individual agents defined in the simulation. The section is 
arranged by agent type, describing the following agents in detail: 


e The bomb detonated in the amphitheater 

e Civilians: Stretcher wounded and those that are “walking wounded” 
or physically unaffected by the bomb 

° Terrorists: Covert agents that incite panic, gunmen _ that 
aggress responders 

e Police: Traffic police, patrolmen, follow-on responders, incident 
commanders, and the SWAT team 

° Firemen from two departments 

e Medical personnel from two different hospitals 


For all agents except the bomb, the author will describe the 
agent class’s activities in plain language, including a diagram that 
highlights transition points between behavior states when necessary (see 
Appendix A for more detail). A description of the Pythagoras modeling of 
the agent class’s behavior follows the plain language discussion. 


50 


2. The Bomb 


The bomb that explodes during the Celebrate Baltimore festival is a device 
approximately the same size as the bomb that destroyed the Murrah Federal 
Office building in Oklahoma City in 1995. It is the equivalent of approximately 
7,350 pounds of ammonium nitrate, in twenty-two 55-gallon drums.” The size of 
the bomb is assumed before the simulation begins and does not change. This 
assumption is required because the terrain in Pythagoras is static. When the 
simulation begins, the terrain is fixed; there is no mechanism by which the 
modeler can enable the strength of the bomb blast to automatically alter the 
terrain. If the modeler sought to analyze different bomb strengths, different areas 


of rubbling must be built. 


a. Cookie Cutter Damage Function 


Damage inflicted on people within the rubbling area is assessed 
using a cookie cutter damage function. There is no partial damage assessment 
resultant from the cookie cutter damage function; an agent within the effective 
radius is killed or not killed, based on the probability of kill associated with that 
weapon. The author chose to use the cookie cutter damage function for 
modeling the close-in damage caused by the bomb to capture the effect of a 
person possibly being shielded by a support beam or other covering object, and 
not being hurt by the blast. It is possible that an agent survives the cookie cutter 
blast effect, but is damaged by the Carleton damage effect outlined below. Note: 
The two blast radii overlap. See Figure 11 for the blast radii associated with the 


cookie cutter and Carleton damage functions. 


°E-mail from Dr. Julie Seton, EPiCS Program Manager, TRAC-WS\MR, titled “Message from 
Dan Edmonson — CT Incident Details for TRAC-Monterey Project,” 24 January 2006, 
office communication. 
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Carleton Blast 
Radius 


Cookie Cutter 
Blast Radius 





Figure 11. Illustration of Blast Radii 
b. Carleton Damage Function 


Damage caused to people outside the rubbling range is assessed 
using the Carleton damage function, a well-documented method for calculating 
damage done by an _ indirect fire weapon. The author chose the 
Carleton damage function to capture the effect of decreased damage to the 
people that are far away from the blast. Figure 12 illustrates the relationship 
between probability of kill (Px) and distance with the Carleton damage function. 
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Figure 12. Probability of Kill Using the Carleton Damage Function 


A large explosion will cause psychological effects, in addition to the 
physical effects felt by people within the blast radius. The psychological effects 
are not limited to people directly (physically) affected by the bomb; panic in a 
large-scale crisis event can be pervasive throughout the area. The technique for 


modeling panic in the civilian population follows in the next section. 


3. Civilians 


In acrisis situation, the civilians involved can be regarded as two separate 
classes: those that can move by themselves (physically unharmed or 
“walking wounded”) and those that cannot (“stretcher wounded’). In this 
simulation, the two are modeled differently. Within the two classes of civilians, 
men, women, boys, and girls are modeled with different characteristics. 
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a. Ambulatory Wounded and Uninjured Civilians 


At the beginning of the simulation, all civilians wander randomly 
through the Celebrate Baltimore area in Baltimore’s Inner Harbor. After the 
explosion of the bomb, civilians that are ambulatory wounded or physically 
unharmed follow the events depicted in Figure 13. 





Civilian Hears Bomb 
Explode, Begins to Panic 












Sustain 
Physical 
Harm? 


No Yes 










Police 
Give Directions 
to Medical 
Station? 












Police Stop, 
Give 1% Aid and 
Directions? 


Civilian Calmed, Moves 
to Medical Station 













Panic Panic 
Continues Continues 


Panic 
Increases 


Terrorist 
Agitator 
Cause More 
Panic? 
















Civilian 
Reaches 
Medical 
Station 





Figure 13. Flow Chart of Uninjured and Ambulatory Wounded 
Civilian Activity 
At timestep one, the bomb explodes in the Baltimore Inner Harbor 
Amphitheater. All of the civilians in the area panic, including those that are 
uninjured, slightly injured, and seriously wounded. If a person is slightly injured, 
one of the emergency first responders must stop, render first aid, and provide 
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directions to the medical triage point that is being set up. Through the interaction 
between first responder and civilian, the civilian is calmed down and begins to 
move to the triage point. If the civilian is uninjured in the bomb blast, their feeling 
of panic is less than someone who was injured. To assuage their feeling of 
panic, a first responder merely needs to spend 30 seconds calming them down 
and providing directions to a triage point at one of three locations outside the 
area affected by the blast. 

Civilians will continue to panic until a first responder provides them 
with the level of comfort that they need. As civilians move to the triage point, it is 
possible that a terrorist agitator encounters them and causes them to panic 
again. If this happens, a first responder must identify the newly panicking 
civilians and calm them again. If a civilian has not been calmed by a 
first responder and a crowd agitator finds them first, it will take the first responder 
longer to calm the civilian and put that person on the path to the triage point. 

(1) Common Qualities. All civilian agents have eyes that are 
instantiated as sensors and voice as a communications device, as previously 
stated. Civilians do not carry weapons. 

(2) Movement Rates. See Table 9 for a translation of 
Pythagoras-specific terms into values that are more familiar to us. The 
movement rates and the sensing information are data derived from AMSAA. 
When calm, civilians move at an average speed of 4 kilometers per hour (kph), or 
at about the pace of an average walk. The maximum speed that a civilian can 
move while calm is 16 kph, or approximately running speed. When panicked, 
civilians have an average speed of 11 kph, with a maximum speed of 22 kph, 
which is a sprint. Appendix A, Section A.1.e. includes a snapshot of the 
spreadsheet used to develop movement rates throughout this research. 
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Value Tolerance Value Tolerance 
Sharaclevsile ——— (Pythagoras) (Actual) (Actual) 
| Movement Rate (Calm)* | 4 koh 12 kph 


I MOVEMEneHas Pane — Rate (Panic)* 11 kph 
Sense and Detect 


Voice 
None 
Vulnerability (Man) —_{ _0.6_} _o___ 
| Vulnerability(Woman) | 75 
| Vulnerability (Boy and Girl) [10 


*Derived aad AMSAA — 





Table 9. Uninjured and Ambulatory Wounded Civilian Characteristics 


(3) Civilian Vulnerability. Vulnerability corresponds to the 
amount of protection an agent has from its environment, e.g., a police officer's 
body armor. In this simulation, men, women, and children have different 
vulnerability levels. The specific assignment of vulnerability levels is not 
scientific. It is a demonstration that it is possible to assign characteristics to 
civilians that can highlight differences between men, women, and children in this 
situation. For this simulation, men have half the vulnerability value of children. 
Women are halfway between men and children. 

(4) Alternate Behaviors. Civilians of this class have two 
alternate behaviors: feeling the psychological effects of the bomb and panic. 

(a) Feeling Psychological Effects of the Bomb. When 
the bomb explodes, civilians hear the bomb and they become afraid. With the 
fear they feel, the civilians also gain a need to be calmed. In simulation, the 
author modeled fear with Pythagoras’s “attribute alpha.” When the bomb 
explodes, every civilian is given 10 units of attribute alpha. Civilians that are 
physically affected by the bomb receive 20 additional units of attribute alpha. 
Ten units of attribute alpha are greater than the threshold that causes panic, so 
the panic behavior is triggered. 

In addition to fear, the explosion awakens a need for 
the civilians to be calmed. In this simulation, calm is modeled by Pythagoras’s 
“resource X.” Civilians are considered consumers of this resource; police, fire, 


and medical personnel are modeled as suppliers of the resource. 
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(b) Panic. When civilians panic, their speed increases 
and they become aware of enemies (terrorists). Civilians want to run away from 
any terrorists they see and will continue to panic until their fear is removed and 
their calm is restored by police, fire, or medical personnel. First responders can 
remove fear (attribute alpha) by giving orders to move toward the triage points. 
These “orders” are a paintball weapon that decrease attribute alpha. In addition 
to taking away fear, first responders are suppliers of calm (resource X). When 
civilians are supplied with more than 25% of calm (resource X) and have less 
than 10 units of fear (attribute alpha), another alternate behavior is triggered: go 
to medical station. 

(c) Go to Medical Station. When civilians cease 
panicking, they move to the nearest of the three medical stations set up outside 
the Celebrate Baltimore area. Civilians will continue to move to a medical station 
unless a terrorist interdicts them and makes them panic by giving the civilians 
additional fear (attribute alpha). 

It is important to understand that the numbers 
associated with attribute aloha and resource X have been chosen to produce 
certain effects, and have no physical significance. The numbers are not as 
important as the relationship between the agents. In this simulation, police and 
terrorists have the same level of effect on civilians. Police are required to interact 
more with civilians that have been physically affected by the bomb. Also, police 
are required to do two things to stop civilians from panicking: they must assuage 


the civilians’ fear and calm them down. 


b. Stretcher Wounded Civilians 


After the explosion of the bomb, civilians that are stretcher 
wounded follow the events depicted in Figure 14. 
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Bomb Explodes 


Medic 
Reached me 
Yet? 










No 







Wait for 
Medic 





Civilian Stabilized, Moves 
to Medical Station 







Civilian 
Reaches 
Medical 
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Figure 14. Flow Chart of Stretcher Wounded Civilian Activity 


At timestep one, the bomb explodes in Baltimore’s Inner Harbor 
Amphitheater. Stretcher wounded civilians remain in one place after the bomb 
explodes; they cannot move until they are stabilized by medical personnel. 
EMTs come to the scene of the explosion from two hospitals. The EMTs 
stabilize the stretcher wounded personnel and move them to the triage point. 
Stretcher wounded civilians are not affected by the panic exhibited by other 
civilians because they are being carried on a litter and cannot otherwise move. 

(1) Common Qualities. All civilian agents have eyes 
instantiated as sensors and voice as a communications device, as previously 
stated. Civilians do not carry weapons. 

(2) Movement Rates. See Table 10 for a translation of 
Pythagoras-specific terms into values that are more familiar to us. The 
movement rates and the sensing information are data derived from AMSAA. The 
average speed that a stretcher wounded civilian can move after being stabilized 
is about 1 kph, or about the speed that a Soldier walks while under a heavy 
combat load. 
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(Pythagoras) | (Pythagoras) | (Actual) | (Actual) 


None 

|Vulnerability(Man) | S05 | 
|Vulnerability(Woman) | 0.75 | 
[Vulnerability (Boy and Girl) [ 40 | 0 


*Derived from AMSAA data. 





Table 10. Stretcher Wounded Civilian Characteristics 


(3) Civilian Vulnerability. In this simulation, men, women, 
and children have different vulnerability levels. The specific assignment of 
vulnerability levels is not scientific. It is a demonstration that it is possible to 
assign characteristics to civilians that can highlight differences between men, 
women, and children in this situation. For this simulation, men have half the 


vulnerability value of children. Women are halfway between men and children. 


(4) Movement Desires. Civilians of this class have one 
desire: move toward the medical triage point. Although the civilians want to get 
to the triage point, they cannot begin moving in that direction until visited by a 
medic. A stretcher wounded civilian’s health is simulated by fuel. When the 
bomb explodes, they run out of fuel, so they cannot move. The medics from the 
hospital are fuel providers. It is only through an interface between the medics 
and the stretcher wounded civilians that the civilians are stabilized (receive their 


fuel) and are able to move to the triage point. 
4. Terrorists 


a. Agitators 


(1) Priority of Work. Terrorist agitators have one goal: incite 
panic in the crowd after the explosion of the bomb. The agitators hide in 
buildings until after the explosion, then come out of the buildings and interact 


with the crowd. The agitators do not focus on individual people, but attempt to 
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cover aS much ground as possible in the response zone, thereby maximizing 
their influence. 

(2) Common Qualities. All terrorist agents have eyes 
instantiated as sensors and voice as a communications device. 

(3) Movement Rates. Movement rates and the sensing 
information are data derived from AMSAA. Terrorists move at an average speed 
of 4 kph, or at about the pace of an average walk. The maximum speed that a 
terrorist can move is 16 kph, or approximately running speed. 

(4) Weapon. Terrorist agitators have a paintball weapon that 
increases the target’s level of attribute alpha, which corresponds to fear in this 
simulation. One shot of the weapon provides enough fear to cause the civilians 
to panic. 

(5) Simulating Covert Operation. By using the ability of 
agents in Pythagoras to change colors, it was possible to enable the agitators to 
“hide” in the crowd. Table 11 illustrates the use of color as a “hiding” mechanism 
for the agitators. At the beginning of the simulation, the agitators consider 
everyone to be an enemy; however, the civilians and police consider the 
agitators to be friendly. As an agitator starts to cause panic, he gradually 
changes color. After five shots, first responders and civilians are able to identify 
the agitator as an enemy. This gradual color change provides the effect that it 


may take the police some time to determine who is causing trouble. 


fp Red | Green _| Blue _| Civilians | Hospital_| Police _ 
Agitator (Start) | 155 | 150 | 0 | Friend | Friend | Friend | 
Agitator (after 1" shot) | 165 | 120 | 0 | Friend | Friend | Friend _ 
Agitator (after 2 shot) | 175 | 90 | 0 | Friend | Friend | Friend _ 
Agitator (after 3 shot) [185 | 60 | O | Friend | Friend | Neutral | 


[Agitator (after 4" shot) | 195 | 30 | 0 | Neutral | Friend | Neutral 
Agitator (after 5" shot) | 205 | 0 | 0 | Enemy | Enemy | Enemy 
Civilians | | 205 | CT *Friend_| Friend | 
[Hospital s/s ss | (S| 220 | 205 | Friend | | Friend | 
[Police | 220215 Friend _| Friend | 


Table 11. Terrorist Agitator Color Change 
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Read Table 11 as follows: The agent in the column sees the agent 
in the row in the manner of the intersecting cell. For example, Civilians see 
Agitator as a friend at the start of the simulation. Civilians see Agitator as neutral 


after shot four. Civilians see Agitator as an enemy after shot five. 


b. Gunman 


(1) Personality and Priority of Work. The terrorist gunman’s 
goal supports the agitator goal of causing panic. The gunman will hide in a 
building until the first responders have arrived on the scene, then attack first 
responders and civilian targets of opportunity. 

(2) Common Qualities. All terrorist agents have eyes 
instantiated as sensors and voice as a communications device. 

(3) Movement Rates. Movement rates and the sensing 
information are data derived from AMSAA. When calm, terrorists move at an 
average speed of 4 kph, or at about the pace of an average walk. The maximum 
speed that a terrorist can move is 16 kph, or approximately running speed. 

(4) Weapon. The gunman carries an AK-47 Assault Rifle. 
Weapon capabilities were derived from data supplied by AMSAA. 

(5) Terrorist in Hiding. The gunman is able to hide by 
programming his base behavior so he is completely concealed, does not move, 
and therefore is undetectable. After the first responders are involved in tending 
to the situation (approximately timestep 375; 25 minutes in actual time), a 
behavior is triggered in the gunman to attack emergency services. During his 
attack, he is visible, armed, and has the desire to attack the enemy, but not get 


too close. This rule of engagement simulates the effect of a sniper. 


5. Police 


This section includes modeling decisions that reflect the author’s 
determination of possible priorities of work for emergency first response 
organizations. Although the author has experience in crisis management 
planning at the military installation level, he established priorities of work entirely 
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without the use of Baltimore standing operating procedures, or consultation with 
Baltimore officials. This simulation is not intended to specifically replicate 
Baltimore emergency response. TTPs may exist that are more effective than 
those established in this simulation. The purpose of this thesis is not to 
demonstrate the author’s knowledge of crisis response TTPs, but rather to show 
that it is possible to model reasonable emergency response procedures in a 
multi-agent environment. 

In this simulation, police forces share similar characteristics. All police 
units except the SWAT team have the same movement rates, sensory ability, 
weapons, and body armor. The author will detail this information for the traffic 
police in Section 5.a., but will not repeat it for subsequent units. Communication 
ability is similar between units. Each unit uses a communication device that 
simulates a handheld radio. The members of each unit can communicate with 
each other, but cannot communicate with members of other units via handheld 
radio. Members of different units can talk to each if they are close enough to use 


the voice communication device. 


a. Traffic Police at Intersections 
(1) General Characteristics. Table 12 describes the 
characteristics of the police that are stationed at the intersections of major roads 
leading to the Celebrate Baltimore area. 
Charactetietic Value Tolerance Value Tolerance 
(Pythagoras) | (Pythagoras) | (Actual) (Actual) 
Movement Rate* 


Sense and Detect* Eyes 


Handheld Radio and Voice 
9mm Pistol, Medical Kit, Orders to Civilians 
Body Armor Level III-A: Most Handguns, Some Small Rifles 


*Derived from AMSAA data. 





Table 12. Characteristics of Traffic Police at Intersections 


(2) Personality and Priorities of Work. Figure 15 illustrates 
the priorities of work for traffic police. 
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Duty at Intersection 









Alert Unit 
Members 
Neutralize 
Enemy Threat 






Maintain Position at 
Intersection 










Provide Directions Provide 1° Aid to 
to Triage Site Civilians in Vicinity 


Figure 15. Priorities of Work for Traffic Police at Intersections 











The author assumes that traffic police are on duty at the 
intersections leading into the Celebrate Baltimore area throughout the festival; 
that is, they are in position when the bomb explodes. In the simulation, traffic 
police maintain position in the immediate vicinity of their intersection, ensuring 
the crisis event is contained and chaos does not spread outside the immediate 
area. If traffic police see an enemy, they will alert the other members of the unit, 
attack the enemy, and neutralize the threat. When the threat is neutralized, the 


traffic police will then return to their initial intersection. 


(3) Weapons and Body Armor. Police carry three tools that 


they can use to influence the agents around them. 


(a) 9mm Semiautomatic Pistol. The 9mm pistol, in 
various forms, is the most commonly carried pistol in police units. The author 
used Army Field Manual 3-23.35 (Combat Training with Pistols, M9 and M11), 
the experience of subject matter experts, and personal experience as a 
military policeman to generate the characteristics of this weapon. Police are able 
to fire one shot using the 9mm every three timesteps, amounting to one aimed 
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shot every 12 seconds of actual time. Maximum engagement range is 
approximately 82 pixels (50 meters). Forty-five rounds is a common standard 
load for police to carry in a large city. A random damage degree of 0.25 means 
that 75% of the damage assessed to a target will be assessed deterministically, 
using the weapon’s probability of kill. Twenty-five percent of the damage 
assessed is stochastic, based on a random number draw. Figure 16 shows the 
weapon’s single shot probability of kill (SSPK), derived by multiplying the 
probability of hit and probability of kill, given a hit. 
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Figure 16. 9mm Single Shot Probability of Kill 


(b) Medical Kit. The medical kit is a “restorative” 
weapon; i.e., its usage increases the target’s health. Police can target unit 
members, friends, and neutral agents with the medical kit. When police use the 
medical kit to perform first aid on a civilian, the policeman is able to reduce the 
level of fear the civilian agent feels (by decreasing the level of attribute alpha). 
For the medical kit to be effective, the administering agent must be within a close 
distance to the target. 


“United States Army, Field Manual 3-23.35, Combat Training with Pistols, M9 and M11, 
June 2003, pp. 1-2. 
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(c) Orders from Police. Orders are a “paintball” 
weapon whose usage causes a decrease in the target’s “fear,” i.e., a decreased 
level of attribute alpha. Police can target friends and neutral agents with the 
orders paintball weapon. Orders are effective to a distance of 100 feet 
(50 pixels). 

(d) Body Armor. In this simulation, all police wear 
body armor that represents armor capable of stopping most handguns and some 
small caliber rifles. This body armor also provides protection against other 
threats, such as blunt force and slashing trauma. Body armor is simulated by 
reducing police vulnerability by 50%. 


b. Police on Patrol 


(1) General Characteristics. Patrolmen share the same 
general characteristics as the police at intersections described in the previous 
section. Patrolmen have one additional capability: they are a supplier of 
Pythagoras’s “resource X,” a measure of influence that will be described in detail 
later in this same section. Each patrol consists of two policemen. 


(2) Personalities and Priorities of Work. Figure 17 is a 
diagram of the patrol areas in which the five patrols operate during the scenario. 
Patrols one and four circulate around their respective areas of operation in 
accordance with the patterns in Figure 17. Patrol number two stays in the area 
of the amphitheater because of crowd density. Patrols three and five have the 


same route, but cover it at different times. 
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Figure 17. Police Patrol Areas 


Figure 18 outlines the patrolmen’s responsibilities and 
priorities of work throughout the simulation. 
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Figure 18. Priorities of Work for Patrolmen 


In the case of police patrolmen, there are several competing 
priorities; the agent must “decide” which priority to address. In Pythagoras, the 
modeler facilitates this decision by assigning appropriate weights to the 
movement desires that correspond with a given priority. In this scenario, the 
patrolman’s top priority is to calm panicking civilians, thereby restoring calm 
throughout the affected area. Tending to civilians that are already injured is of 
secondary importance. By calming the panicking crowd, the patrolman helps to 
ensure that more civilians are not injured. Enemy engagement is of tertiary 


importance because the SWAT team is alerted to engage the threat. 


In this simulation, panicking civilians have two 


characteristics that alert the police of their state of panic: 
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e Low level of calm (Pythagoras’s resource X) 


e Raised level of fear (Pythagoras’s attribute alpha) 


Police patrols alleviate both of the above civilian conditions, enabling them to be 
relieved of their state of panic and move to the medical triage point. 


Patrolmen are set up in Pythagoras as suppliers of 
resource X. As stated earlier, resource X in the simulation represents an 
objective measure of an agent’s level of “calm.” The Pythagoras model enables 
consumers of a given resource to notify suppliers when the consumer reaches a 
certain threshold, established in advance by the modeler. Suppliers receive the 
resource request from consumers and move to the consumers to impart the 
resource needed. In this simulation, the calmed civilian is given enough “calm” 
for the entire scenario, if not attacked by a terrorist. If the civilian is attacked by a 
terrorist, that civilian will panic again. Reversion to the panic behavior causes a 
loss of the resource, reinitiating the call for more. Patrolmen regenerate their 
supply of resource X; thus, they will not run out of the ability to calm civilians. 


In addition to the ability to calm civilians using resource X, 
the patrolmen provide directions to a medical triage site and orders to go there. 
These orders are modeled as a “paintball” weapon, an influence tool that 
reduces the civilian’s level of attribute alpha (fear). A civilian is considered calm 
as long as the civilian: 


e Maintains a resource X level greater than 25% of maximum level 


e Has an attribute alpha level lower than the amount imparted by one 
terrorist interaction 


If there are no panicking civilians, the patrolman will attempt 
to identify wounded civilians. Upon identification of a wounded person, the 
patrolman will move to the wounded person and perform first aid. In this 
simulation, first aid is modeled with an influence tool that has restorative 
properties. When engaged with this influence tool, the target’s health increases. 


After the target’s health is restored, the patrolman will again try to identify if a 
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civilian needs to be calmed. If not, the patrolman will seek to identify another 
wounded civilian. 

Patrolmen will engage an enemy that they see if and only if 
their primary and secondary missions have been fulfilled. If the situation is such 
that the patrolman must engage an enemy threat, the weapon of choice is the 
patrolman’s 9mm pistol. After the engagement, the patrolman will again check to 
see if a primary or secondary mission must be fulfilled. If not, and there are no 
additional enemies in sight, the patrolman will continue along the predetermined 
patrol route. 


Cc. Follow-On Police 


(1) General Characteristics. Follow-on police are police 
personnel that assemble at the Central District headquarters in response to the 
first alarm initiated by Baltimore’s central dispatching authority. The 
characteristics of these personnel are the same as the traffic police and the 
patrolmen that were on station when the bomb exploded. Follow-on police enter 
the simulation as two teams of three policemen each. These teams arrive at 
intervals of 15 minutes (225 timesteps) after the explosion, starting at 
timestep 225. 


(2) Priorities of Work. First-alarm police have similar goals 
to the patrolmen described in the previous section; the decision template 
diagram of Figure 18 applies. The follow-on patrols seek to spread a calming 
influence by clearing the buildings in the area surrounding the bomb blast and 
directing the people to the medical triage points. Figure 19 identifies one route 
that first-alarm police follow to provide assistance to the area affected by the 
bomb. The route shown illustrates a west to east path that is taken by one team 
of three personnel. The other team of three that is dispatched at the same time 
follows the same path, but in an east to west direction. 


69 


Initial 
Location 





Figure 19. West to East Route of Follow-on Police 


d. Incident Command Post (ICP) 


(1) General Characteristics. The ICP is located in the 
convention center, directly to the west of the amphitheater. The purpose of the 
ICP is to provide situational awareness to the entire response force. It is through 
the ICP that the different organizations involved in the response know what each 
other are doing. The ICP is comprised of one leader from each of the 


organizations involved in the simulated response to this crisis: 


e Traffic Police 

° Initial Response Patrolmen 

e Fire Department 

e Hospital 

° Central District Police (first-alarm, follow-on responders) 


e SWAT Team 


(2) Incident Command Structure. See Figures 20-22 for 


diagrams of the ICP’s functionality. 
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Figure 20. ICP Receives Information 





Figure 21. Information Sharing 
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Figure 22. ICP Disseminates Information 


The leaders that comprise the ICP receive information from 


the members of their individual organizations, using a communication device that 
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simulates a standard handheld radio. The ICP is a well-connected organization, 
meaning that each member of the ICP has communication with every other 
member. Members of ICP communicate internally via a communication device 
that simulates a voice. After information is passed throughout the ICP, the 
leaders send the consolidated information back to the members of the 
organizations that are conducting the crisis response operations. In this manner, 
every member of every organization has situational awareness of the 


operational environment. 


One example of the importance of the ICP is the 
identification of the terrorist gunman in this scenario. If a patrolman encounters 
the gunman, he is at a disadvantage; the patrol is armed with a pistol, the 
gunman with an AK-47. The patrolman communicates the location of the 
terrorist to the ICP. The ICP transmits that location to the SWAT team, which 
responds with nine personnel carrying M4 carbines, a decided advantage for the 


good guys. 


e. SWAT Team 


The SWAT team is located in the Central District headquarters, 
north of the bomb attack location. The mission of the SWAT team in this 
simulation is to receive information about the location of terrorists, move to that 
location, and neutralize the terrorist threat. After the threat has been neutralized, 
the SWAT team returns to the Central District headquarters to await another call. 
The team is comprised of nine police officers. These officers have sensory and 
communication characteristics that are similar to the other police officers 
described in this simulation. Table 13 outlines the characteristics of SWAT team 
members. SWAT team members have the capability to move faster than other 
police officers and they are more heavily armed. SWAT members carry a rifle in 
addition to their service pistol. 
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ewes Value Tolerance Value Tolerance 
Pnarari enone (Pythagoras) | (Pythagoras) | (Actual) (Actual) 


Movement Rate’ 12 kph 
rr 2 eee 


Eyes 
Handheld Radio and Voice 
M4 Carbine, 9mm, Medical Kit 


*Derived from AMSAA data. 





Table 13. Characteristics of SWAT Team 
f. Firefighters 


(1) General Characteristics. Firemen are initially located at 
two fire stations. Fire Station 23 is located approximate eight-tenths of a mile 
east of the location of the bomb blast. Fire Station 33 is located about 
three-quarters of a mile south of the blast. Table 14 includes the general 
characteristics of the firefighters in this simulation. In addition to the ability to 
provide first aid with the medical kit and direct civilians to the triage point, 
firefighters have the ability to stabilize stretcher wounded personnel and move 
them to the triage point. The mechanism for this action is described in detail in 


Section IV.A.5.g, which discusses medical personnel. 


eat Value Tolerance Value Tolerance 
Geraci ean (Pythagoras) | (Pythagoras) | (Actual) (Actual) 


Movement Rate’ 
po Eyes 


Eyes 
Medical Kit, Orders to Civilians 
None 


*Derived from AMSAA data. 





Table 14. Characteristics of Firefighters 


(2) Priorities of Work. Firefighters deploy to the bomb attack 
area five minutes (75 timesteps) after the explosion occurs. Teams of 
five personnel deploy from their respective fire stations directly to the two 
buildings that sustain rubbling damage. Fire Station 23 moves to the northern 
rubble area. Fire Station 33 moves to the southern rubble area. The firefighters’ 
mission is to search the rubble for survivors of the blast. Upon identification of 
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survivors, the firefighters have the capability to provide the level of medical 
support that a survivor needs, from first aid (using the medical kit, as described 
earlier) to stabilization of stretcher wounded civilians (described in detail with 
discussion of hospital personnel in the next section). Upon stabilizing the 
civilians, firemen will order them to a triage point. Through the interaction 
between firefighters and civilians, calm is restored and panic reduced in the 


same manner as with police interactions described earlier. 


g. Medical Personnel 


(1) General Characteristics. Medical personnel are initially 
located at two hospitals. Mercy Medical Center is located approximately 
nine-tenths of a mile north of the location of the bomb blast. Saint Johns Hopkins 
Hospital is located about two miles northeast of the blast. Table 15 includes the 
general characteristics of the medics in this simulation. In addition to the ability 
to provide first aid with the medical kit and direct civilians to the triage point, 


medics have the ability to stabilize stretcher wounded personnel and move them 


to the triage point. 


ae Value Tolerance Value Tolerance 

Characteristic (Pythagoras) | (Pythagoras) | (Actual) (Actual) 

27 kph 14 kph 
Movement Rate (Treatment) | 1 | | OS kph [OOO 
Pye sd 


Eyes 
Handheld Radio and Voice 
None 


*Derived from AMSAA data. 





Table 15. Characteristics of Medical Personnel 


(2) Treatment of Stretcher Wounded Civilians. After the 
bomb explodes, many civilians in the immediate area of the blast are critically 
wounded. Their wounds are serious enough that they cannot be stabilized by a 
policeman with standard shift equipment. In this simulation, only a trained agent 
(from a hospital or fire department) has the training and equipment needed to 
stabilize a stretcher wounded civilian. When an EMT identifies a stretcher 
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wounded civilian, the EMT moves to the civilian and provides treatment. 
Following the treatment, the civilian is moved to the triage point. 

The author uses Pythagoras’s concept of fuel to simulate the 
treatment of people who are unable to move due to their injuries. The use of fuel 
is similar to the police force’s use of resource X to restore calm in panicking 
civilians. Stretcher wounded civilians cannot move because they are consumers 
of fuel and have run out. In this simulation, only EMTs are suppliers of fuel, so 
only EMTs can effectively treat stretcher wounded civilians. When a 
stretcher wounded civilian is short on fuel, that agent will request fuel from the 
supplier. The supplier will move to the consumer and provide the fuel. At that 
time, the consumer will begin moving in accordance with their desires. 

There is no mechanism in Pythagoras to simulate one agent 
carrying another. After being stabilized, the stretcher wounded civilians move to 
the medical triage point under their own power; the speed is comparable to a 
soldier who is moving under heavy combat load. Speed under heavy combat 
load (2 kph) is approximately the speed that trained personnel would carry a 
stretcher. To simulate the EMT carrying the stretcher, the speed of the EMT is 
reduced to only 0.5 kph. If the EMT was actually carrying the stretcher, that 
agent would not be able to provide treatment to another patient until the first 
patient was dropped off and the EMT returned to the scene. This effect is 
accounted for in the simulation because the EMT’s movement is slowed 
dramatically; i.e., due to the slow movement, the EMT has an appropriate lag 
time between treatments. 

It is well known, well established, and often repeated that “all 
models are wrong, but some are useful.”*’ The previous pages have been a 
discussion of how the author simulated several effects that capture the essence 
of emergency first response to a crisis event. There are some limitations to this 
model and modeling technique. The next section discusses the limitations 


“George Box, “Robustness in the Strategy of Scientific Model Building,” in Robert Launer 
and Graham Wilkinson (Eds.) Robustness in Statistics, 1979, p. 202. 
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identified, the artificialities associated with this type of simulations, and the 
assumptions made to work around those artificialities. 


B. MODEL LIMITATIONS AND ARTIFICIALITIES 
1. Model Limitations 


It is important to note that the Pythagoras model is currently a 
beta version. The developers of the model have been particularly helpful in 
trying to provide a quality model to use for this thesis and others that 


are ongoing. 


a. Invulnerable Agents 


As mentioned previously in the discussion of timestep selection, 
agents jump from pixel to pixel with each timestep. It is possible for an agent to 
jump into a terrain feature from which the agent cannot move. For example, if an 
agent is standing next to a feature with movement factor 0 (e.g., a wall) and that 
agent has a desire to move past the wall, that agent may jump into the wall. The 


agent can no longer move because the feature’s movement factor is 0. 


This problem is especially noticeable in the event that a terrorist 
being tracked by a police patrolman jumps into a wall. The patrolman sees the 
terrorist’s last position and sees the place at which the terrorist went into the wall. 
The patrolman will continue to engage the terrorist, even though he cannot see 
him, due to the wall having a concealment factor of 1.0. Since the wall has a 
protection factor of 1.0, the terrorist cannot be killed. The result is that the 
patrolman remains at the wall until the end of the simulation. In addition, the 
patrol communicates with his unit members, relaying the enemy’s location, 
resulting in the entire unit being stationed at the wall, shooting ineffectively at the 
wall, for the rest of the scenario. This problem does not occur frequently enough 


to impact the analysis. 
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b. Use of Fuel 


The modeler can use the concept of fuel in Pythagoras in several 
ways, one of which is actually using it as fuel. One limitation of Pythagoras is the 
mechanism by which fuel is passed from the supplier to the consumer. The 
supplier provides all of the necessary fuel in one timestep and then moves to the 
next consumer. This situation is an accurate representation of the 
supplier/consumer relationship if the supplier is dropping off a bulk fuel package. 
There is no ability for a modeler to represent fuel being passed from supplier to 
consumer over time, e.g., pumped at a certain rate of units per timestep. In this 
simulation, fuel represented health. If there was a capability to simulate a rate of 
fuel transfer, the EMTs stabilizing stretcher wounded civilians would have 
needed to stay directly with their patient until sufficient fuel was provided. 


This deficiency extends to the use of resources X, Y, and Z, which 
are modeled in the same manner as fuel in Pythagoras. The supplier/consumer 
exchange deficiency is reported to have been fixed in Pythagoras version 1.10.1, 
but this has not been verified by the author. 


Cc. Weapon Usage 


There are situations in the current simulation in which agents are 
targeted with weapons they should not be targeted with. For example, police are 
supposed to target only enemies with the 9mm pistol. Civilians are considered 
friends to the police force. Unfortunately, situations occur in which police target 
civilians with the 9mm and kill them. This occurrence is (fortunately) relatively 
rare and does not provide a significant impact on the Measures of Effectiveness 
(MOEs) of this thesis. 


There are instances within the simulation in which weapons that 
are restorative actually kill. These isolated instances occurred when police used 
their orders, a paintball restorative weapon, to influence civilians. This 
occurrence is also rare and does not impact the analysis of MOEs in 
this research. 
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d. Injured Friend Identification 


Currently, agents do not react appropriately using the move 


“toward injured friend” logic. In fact, this desire does not appear to work at all. 


e. Identification of Undetectable Agents 


Prior to timestep 375, the terrorist gunman is in a state that sets his 
detection factor across all sensing bands to be 0.0. Within the vignette, there are 
several instances in which the terrorist gunman is identified when he should be 
completely undetectable. This model deficiency did not impact analysis because 
as the gunman was identified, appropriate action took place, i.e., location was 
communicated to friendly units, the friendly units consolidated and killed 
the terrorist(s). 


f. Error Reporting 


Error reporting in Pythagoras is not user friendly. If there is an error 
in the Pythagoras Java code, a generic error message appears on the screen 
and the scenario file will not open. The next step for the user to follow is to open 
the .xml file in an xml editor and look for errors. The user is not given an 
indication where to look in the code for the error. Heuristics exist to guide the 
user in looking through the code, but often fail. There are times that a working 
file is compared to a nonworking file and the differences should not cause errors 
in running the file. It would be useful for the developers of Pythagoras to provide 
an error message that directs the user to the specific area of computer code that 


fails in a model run. 
2. Artificialities 


a. Terrain 


(1) Rubbling. The rubbling in this simulation was created 
without the use of scientific data. The author created the rubble to capture the 
effect of firefighters moving to the area and searching. 
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(2) Bomb Damage. In reality, a 7,300-pound bomb would 
have damaged more than two buildings. The desired effect of this simulation 
was to capture first responder interaction with the civilian crowd that was close to 
the explosion. This effect was captured in a satisfactory manner without detailed 
research into building structure thresholds. 


(3) Triage Points. Due to the static nature of the Pythagoras 
terrain, triage points had to be set up before the simulation started. The 
assumption that the author made in using prepositioned triage points is that the 
location of these points was establish in pre-event planning. 


b. Civilian Effects 


Not addressed in this simulation are the unpredictable effects of 
crises on humans. Inevitably, there are civilians who will try to perform first aid 
on wounded people, help carry stretchers, and seek the opportunity to help the 
first responders. Also apparent, with the events surrounding the response to the 
2005 hurricanes, is that some people will take advantage of the situation by 
becoming unruly, or even engaging in criminal activity. Neither effect is 


simulated in this vignette. 


This chapter provided a detailed, step-by-step discussion of the 
technique used to model emergency response to a crisis in a MAS. The 
modeling described here corresponds to steps 8 and 9 of the methodology 
outlined in Figure 6 and subsequently discussed. The following chapter details 
steps 10 and 11 of Figure 5, identifying factors of concern and their levels. 
Chapter V will also discuss the experimental design process—the bridge 


between implementing the model and analyzing the data. 
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V. EXPERIMENTAL DESIGN METHODOLOGY 


This chapter describes steps 11 and 12 of the simulation methodology 
depicted in Figure 6, outlining the design of experiments (DOE) for this research. 
The DOE is the mechanism the analyst uses to vary input factors, conditioning 
the model to answer questions of interest; it is the bridge between model 
development and analysis of data. The author will describe the three different 
experimental designs chosen to collect the data needed to analyze the TOPOFF 
scenario. Following the discussion of the DOE and input factors, the author will 
describe the chosen MOEs that answer the questions posed at the beginning of 
this research. The chapter concludes with a brief discussion of tools and 
techniques that support the development of the DOE and the analysis of the 
resultant data. 


A. METHODOLOGY 


The power of simulations, especially MAS, results from the ability of an 
analyst to quickly analyze the consequences of decisions. In the case of 
emergency response, the decision may include the mix of response personnel 
and the time at which they are committed. The decision may also be less 
tangible, such as a recommendation on TTPs for a response organization. 
Gaining insight into these questions requires testing the given scenario by 
varying specific factors of interest over a range of values. This model, or another 
model that seeks to emulate human behavior, is stochastic; i.e., it involves 
variability. The stochastic nature of this model results in the necessity of running 
multiple replications of each unique model setting to obtain an empirical 
distribution of the range of possible responses. 

The number of possible variables in this simulation is staggering; it 
highlights the need for an efficient DOE. There are three “sides” in this 
simulation: civilians, first responders, and terrorists. See Table 16 for a 


breakdown of the total number of unique agents in this simulation. 
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. . Number of | Total Number of 


Walking, Stretcher Wounded| 2s] 
First Responder Police, Fire, Medical 
Agitator, Gunman 


Table 16. Agent Side Breakdown 





Each unique agent class has several qualities that can be varied: 


e Number of agents 

e Vulnerability of agents 

e Agent marksmanship (police and terrorist) 
e Agent communication effectiveness 


In addition to varying individual agent qualities, it is possible to vary overall 
qualities associated with several agents at once. For example: 
e Effectiveness of a weapon in general, 12 weapons in the simulation 
(versus marksmanship of an individual agent using that weapon) 


° Probability of successful communication of a given piece of 
equipment, nine different communication devices (versus an 
individual’s effectiveness in using the device) 


e Probability of detection for the sensor 


From above, the total possible number of factors that can be varied in this 


simulation is 258. 


59 unique agents x 4 factors per agents = 236 factors 
+ 12 different communication devices = 12 factors 
+ 9 different communication devices = 9 factors 
+ 1 sensor = 1 factor 


Of the 258 possible factors, the author determined that varying 48 factors 
would provide the data to answer the research questions posed. Using a simple 
grid design, varying these 48 factors at three levels (a high level, medium level, 
and low level) would result in 3“° (7.98 x 10”) unique model runs or design 
points. Each model run takes approximately 40 minutes, resulting in a need for 
5.32 x 10 CPU hours. This simulation is a stochastic simulation, meaning that 
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the analyst must run several replications of each design point to achieve 
meaningful results. Assuming that 30 replications are sufficient (many 
simulations require 50 or 100), the entire simulation would require 1.60 x 10°4 
CPU hours, which is about 116 trillion times the approximate age of 
the universe.” 

Given that we do not have that long to wait, it is necessary to develop a 
much more efficient DOE than the traditional gridded design, or consider far 
fewer factors. The author chose to find an efficient design. Robust, efficient 
designs of experiments enable analytical factor selection, causing the resulting 
analysis to be less affected by personal judgments and bias. 

The search for an efficient DOE quickly led to work done by 
U.S. Army Lieutenant Colonel Thomas M. Cioppa, using NOLHs. As mentioned 
earlier, data farming provides a method to take advantage of efficient DOEs, 
such as the NOLH, and supercomputers to grow an abundance of data points for 
further exploration. The NOLH design efficiently searches the high-dimensional 
input space defined by an intricate response surface. The NOLH has the 
following characteristics:* 


e Approximate orthogonality of all input factors 

e A collection of experimental cases representative of the subset of 
points in the hypercube of explanatory variables (space filling) 

° Ability to examine 20 or more variables efficiently 

e The flexibility to analyze and estimate multiple effects, interactions, 
and thresholds 

e Requires minimal a priori assumptions on the response 

e Easy design generation 

e Ability to gracefully handle premature experiment termination 





“age of the Universe. Retrieved on 19 May 2006 from the World Wide Web at 
http://en.wikipedia.org/wiki/Age of the Universe, using NASA's Wilkinson Microwave Anisotropy 
Probe (WMAP). 


““Thomas M. Cioppa, “Efficient Nearly Orthogonal and Space-Filling Experimental Designs 


for High-Dimensional Complex Models,” Ph.D. Dissertation, Operations Research Department, 
Naval Postgraduate School, Monterey, CA, 2002. 
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Refer to Cioppa’s dissertation for additional information regarding a 
NOLH. Cioppa’s algorithm requires only 2,049 design points to analyze the 
same solution space addressed by the previously mentioned gridded design! 
Instead of 3*° runs, we require 2''+1 runs, saving orders of magnitudes in 
computing time. With this design, it is possible to execute the DOE with a 
sufficient number of replications to achieve statistical relevance, using current 
technology, in a reasonable amount of time. See Appendix B (Design of 
Experiment Modeling) for detailed information about calculating the number of 


runs required. 


Cioppa’s is an _ excellent design, but current work ongoing by 
U.S. Army Lieutenant Colonel Alejandro Hernandez extends the ideas developed 
by Cioppa. Hernandez uses a method incorporating FRLHs to create 


experimental designs that are even more efficient than Cioppa’s. 


Hernandez found that using correlation reduction methods, it is possible to 
create a design that meets Cioppa’s criteria for near orthogonality when the 
number of design points is greater than or equal to 49. In fact, the number of 
design points need only be greater than or equal to about three times the number 
of variables to maintain Cioppa’s criteria of near orthogonality, see Figure 23. 


number of design points (levels) 
number of variables 


When n= 49, 
n= 3k 





Figure 23. Relationship Between the Number of Variables and the Number of Levels in 
Hernandez’s Design of Experiments 


The result of Hernandez’s work is the requirement for only 144 design 
points to complete a nearly orthogonal main effect design of experiments that will 
provide insight into the behavior of this problem’s response surface, in 
48-dimensional space. This 144 x 48 design represents a possible savings of 


over 93% in terms of design points and required computing power! 
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The benefits that result from using Hernandez’s design come with costs 
that should be understood. This design is less space filling than designs using 
Cioppa’s and Ye’s* NOLH and OLH designs. In addition, the designs by 
Cioppa and Ye guarantee orthogonality or near orthogonality with the inclusion of 
one interaction or one quadratic term. Hernandez does not yet make that 
guarantee with his designs. 


B. EXPERIMENTAL DESIGNS 
1. Flexible Random Latin Hypercube Design (FRLH) 


The author completed two separate experiments using the FRLH design. 
For both experiments, the hypercube generated by Hernandez was crossed with 
a simple 3 x 1 grid that captures three different probabilities of communication. 
The probabilities of communication examined are: 100%, 75%, and 50%. At 
100%, a piece of information successfully passed between agents is certain to be 
received. At a level of 50%, the piece of information is passed with probability 
one-half. See Tables 17-19 for the factors varied in both FRLH experiments. 


Level | Level Given Factor 
(Integer)* first responder to civilian ratio 


Civilian Vulnerability 1 Identify effect of increased civilian 
(Continuous) susceptibility to terrorist action 


Desire of Civilians to 1 Examine effect of civilian fear 
Avoid Terrorists of terrorists 
(Continuous) 


*12 correlated factors. 





Table 17. Civilian Factors Varied in FRLH Design 


“K.Q. Ye, “Orthogonal Column Latin Hypercubes and their Application in Computer 
Experiments,” Journal of the American Statistical Association — Theories and Models, Vol. 93, 
No. 444, pp. 1430-1439, December 1998. 
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Level | Level Given Factor 
Location 1 (Integer) police ratio; key terrain for terrorists 
Location 2 (Integer) police ratio; key terrain for terrorists 
Location 3 (Integer) police ratio; key terrain for terrorists 
Location 4 (Integer) police ratio; key terrain for terrorists 


acronis 
(Integer) police ratio; key terrain for terrorists 
final il eda 
Continuous 
some | ° [1 srenrameneo snr 
Gunmen (Continuous 
og [OT eT eermenee 
Continuous inciting panic 
seecece | ° | bgttnceeensme 
Gunmen (Continuous forces 


Lethality of Bomb (Integer) | 0 | 10 __ [Identify possible effects of bomb strength 


Table 18. Terrorist Factors Varied in FRLH Design 


Level | Level Given Factor 
St. Johns Hopkins (Integer) key terrain for first responders 
a 
Mercy Medical (Integer key terrain for first responders 
Patrol Area 1-5* (Integer) key terrain for first responders 
at 15 minutes (Integer) response 





at 30 minutes (Integer) response 

at 45 minutes (Integer) response 

Areas 1-12** (Integer) key terrain for first responders 

SWAT Team (Integer) posture (waiting for call to attack) 
eae” | * eters 
(Continuous) (against terrorist, for civilians) 


*B separately altered factors; **12 separately altered factors. 





Table 19. First Responder Factors Varied in FRLH Design 
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There are two criteria that Cioppa uses to assess the orthogonality or 
near orthogonality of a design matrix comprised of each input variable (column) 
and design point (row). These two criteria are the absolute maximum pairwise 


correlation and the design matrix condition number.” 


The absolute maximum pairwise correlation is computed by calculating the 
correlation between every pair of input columns in the design matrix, then taking 
the largest absolute value. A value of 0 is best, representing an orthogonal 
design matrix. A value of 1 is worst, meaning that at least 1 column is perfectly 


correlated with another column. 


The condition number of the design matrix is used to “examine the 
sensitivities of a linear system.” The condition number can also reveal the 
degree of nonorthogonality of a given design matrix. The best value possible for 
condition number is 1, associated with an orthogonal design matrix. 
Nonorthogonal design matrices have condition numbers greater than 1, meaning 
that the design matrix has a degree of multicollinearity. 


The designs created by Hernandez for this research meet Cioppa’s criteria 
for near orthogonality. One design consists of a 144 design points, and is 
described in detail in Section V.B.1.a. This design has an absolute maximum 
pairwise correlation of 0.008 and a condition number of 1.046. The second 
design is detailed in Section V.B.1.6. This design, consisting of 1,008 design 
points, has an absolute maximum pairwise correlation of 0.001 and a condition 
number of 1.006. 


a. Small FRLH Experiment 


The first experimental run fully capitalized on the efficiency of 
Hernandez’s FRLH design. This experiment involves the use of Hernandez’s 
““Thomas M. Cioppa, “Efficient Nearly Orthogonal and Space-Filling Experimental Designs 

for High-Dimensional Complex Models,” Ph.D. Dissertation, Operations Research Department, 


Naval Postgraduate School, Monterey, CA, 2002. 


“SE-mail from LTC Hernandez titled “Roginski New Designs,” 11 May 2006, office 
communication. 
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“base” design, using 144 design points, allowing for the possible use of 
43 numeric factors and 5 qualitative factors (48 total factors). Based on a final 
analysis of the factors to be analyzed, the author used 37 of the 43 unique 
columns, and did not use the five qualitative factors. The effect of this decision 
on the design’s maximum pairwise correlation and condition number was to drive 


them lower, making the design even more nearly orthogonal. 


The total number of runs in the small FRLH experiment is 
144 x 15 x 3 (each level of communications grid), or 6,480 runs. The advantage 
of this experiment is its relatively short run time, when compared to the design 
discussed in the next section. The author received data back from the MHPCC 
in three days, versus a three week turn around time for the large experiment. 
The run resulted in approximately 4,320 rows and 5,310 columns of raw data for 


analysis. 


The disadvantage of this design is that its sparse coverage of the 
solution space results in a much more granular understanding of the response 
surface. In situations which the response surface is complex, important detail 


may be missing from the data. 


b. Large FRLH Experiment 


The second experiment run was the largest, combining the 
efficiency of Hernandez’s FRLH design with the space filling nature of 
Cioppa’s design. The same 37 variables used in the previous experiment are 
used in this experiment; a heuristic approach was used to settle on 1,008 as the 
number of total design points. 


The advantage of using this approach is the space filling nature of 
using over 1,000 design points. This number of design points enables much 
better coverage of the solution space than the 144 design point experiment, 
previously discussed. Greater coverage of the overall solution space results in a 
better understanding of the response surface, and less of a chance to miss an 
important result. The disadvantage of this design is that, while relatively efficient 
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compared to a gridded design, this experiment still takes longer to run than the 


smaller FRLH design. 


Using the output of previous runs, the author determined that 
15 replications of each design point are sufficient for analysis of data. The total 
number of runs in the large FRLH experiments is 1,008 x 15 x 3 (each level of 
communications grid), or 45,360. This simulation requires 40 minutes of 
CPU time for each run; therefore, the expected length of this experiment was 
30,240 CPU hours. The set of runs was completed with the help of the MHPCC 
supercomputers in about three weeks, resulting in approximately 45,360 rows 
and 5,310 columns of raw data for analysis. 


2. Gridded Design 


The author also used a gridded design, again run at the MHPCC. The 
intent of this design is to focus directly on a few factors, while holding the rest 
constant. The author considered the following variables and levels in this design 
(see Table 20). The total number of runs in the gridded experiment is 
5° (three variables at five levels each) x 20 replications, or 2,500 runs of the 
simulation. The expected time for 2,500 runs at MHPCC was about 1,666 CPU 
hours. The author received data back from Maui in two days, versus a 


three-week turn around time for the large experiment. 
Factor Nanie Low | High Number | Desired Outcome From Varying 
Level | Level of Levels Given Factor 
Probability of 0.1 0.2 5 Identify effect of communication 
Communication equipment on response 


Fd 
Traffic Police* police to terrorists, and to patrols 
Patrolmen** to terrorists, and traffic cops 

“Each traffic location is given the same number; **Each patrol location is given 

the same number. 





Table 20. Gridded Design Completed at MHPCC 
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3. Measure of Effectiveness 


Analyzing a data set that is 45,000 rows by 5,300 columns or even 
6,000 rows by 5,300 columns, is a bewildering task if the analyst lacks focus and 
scope. MOEs provide the scope and focus that the analyst needs to provide 
timely support to the decision maker. The analyst establishes MOEs that help 
provide insights and answers to research questions posed at the beginning of a 
given project. To understand the use of MOEs, it is important to understand the 
difference between MOEs and measures of performance (MOPs). Sections a 
and b below show the definitions of a MOP and MOE from the Defense Modeling 
and Simulation Office (DMSO). 


a. Measure of Performance, Defined” 


A measure of how the system/individual performs its functions in a 
given environment (e.g., number of targets detected, reaction time, number of 
targets nominated, susceptibility of deception, task completion time). It is closely 
related to inherent parameters (physical and structural), but measures attributes 
of system behavior. 


b. Measure of Effectiveness, Defined” 


A qualitative or quantitative measure of the performance of a model 
or simulation or a characteristic that indicates the degree to which it performs the 
task or meets an operational objective or requirement under specified conditions. 


Cc. Use of MOEs 


MOPs determine how well one small entity fared in the mission. 
MOEs identify how well the mission was accomplished overall.“? Establishing 





“’DMSO, definition of Measure of Performance. Retrieved 21 on May 2006 from the World 
Wide Web at https:/Awww.dmso.mil/public/resources/glossary/results?do=get&search_ text=MOP 


“8DMSO, definition of Measure of Effectiveness. Retrieved on 21 May 2006 from the World 
Wide Web at https:/Avww.dmso.mil/public/resources/glossary/results?do=get&search_ text=MOE 


““Donald E. Brown, Ph.D., and C. Donald Robinson, “Development of Metrics to Evaluate 
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defensible MOEs is much more of a challenge than establishing interesting 
MOPs, especially in an area that may directly impact peoples’ lives, such as 


emergency response. 


For example, baseball fans compare their favorite players’ batting 
average, on base percentage, and number of runs batted, to try to determine the 
better player. These are measures of the players’ performance, not 
effectiveness. To measure a player's true effectiveness, we must ascertain how 
much that player contributes to winning the game, winning the division, or the 


World Series.°° 


As stated previously, MOEs are established to answer questions 
posed at the beginning of research. Below are the four MOEs selected for this 
research, followed by Table 21 that highlights the research questions associated 
with each MOE. The MOEs address the effectiveness of the first response 
agencies in mitigating the effects of the crisis event, seeking to achieve the 


following goals: 


e Stabilize wounded civilians 

e Restore calm in the area 

e Eliminate further threats 

e Maintain safety of first responders 
MOEs 


e MOE 1. The first MOE is the percentage of civilians that have not 
received sufficient medical treatment by the end of the first hour. 
This MOE directly addresses the first priority established above by 





Effectiveness of Emergency Response Operations,” presented at the 10" International Command 
and Control Research and Technology Symposium, The Future of C2, June 2005. Retrieved on 
2 December 2005 from the World Wide Web at www.dodccrp.org/events/2005/1 0th/papers/ 


326.pdf 


Brian G. McCue, Christine A. Hughes, and Kathleen M. Ward, “Analysis Planning for 
Domestic Weapon-of-Mass-Destruction Exercise,” The CNA Corporation, The Occasional Paper 
Series, (IPR) 10856, May 2003, pp. 21-22. Retrieved on 2 December 2005 from the World Wide 
Web at www.cna.org/documents/IPR10856_1.pdf 
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measuring the effectiveness of first responders in the 
“Golden Hour,” or the first hour after the bomb blast. 


e MOE 2. The second MOE is similar to the first in that it measures 
the effectiveness of the first responders. The second MOE is the 
percentage of civilians at the triage points at the end of the 
first hour. Civilians only go to the triage points after first responders 
have successfully interacted with them, provided aid, and assuaged 
their feeling of panic. This MOE highlights the success of 
first responders in achieving the second priority outlined above. It 
measures the first response force’s psychological effectiveness on 
the civilians, where the first MOE measures physical effectiveness. 


e MOE 3. The third MOE is the means that the analyst will use to 
determine success in achieving the third priority. The third MOE is 
the percentage of times among the replications for a given design 
point that all terrorist gunmen were neutralized. Total neutralization 
of the gunmen means there is no further lethal threat. 


e MOE 4. The fourth MOE is the percentage of first responders 
injured or killed during response operations in the first hour. Using 
this MOE, the analyst can gain insight into the factors that are most 
important in determining survivability of first responders. 


Research Question | MOE | 


What is an appropriate methodology for use of a MAS Figure 6, 
environment in the modeling of emergency response to the Step 10 
simulated VBIED that explodes near the amphitheatre in 


Baltimore’s Inner Harbor? 


What is the most appropriate mix of police, fire, and 
medical assets? 
for emergency response to VBIED in Baltimore’s Inner Harbor? 


Table 21. Research Questions and Associated MOEs 





C. EXPERIMENTAL DESIGN TOOLS AND TECHNIQUES 


Visual observation of the Pythagoras model provides a certain degree of 
value; however, the purpose of a MAS, such as Pythagoras, is to view a 
problem’s high-dimensional space. People can see in three dimensions. With a 
degree of chart wizardry, one can argue that it is possible to gain an idea of 
four-or five-dimensional space. The response surface associated with this 
problem is in 40+ dimensional space, requiring the use of several different tools. 


This section describes those tools used in this research. 
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The tools bridging Pythagoras-produced data to the analysis conducted 
include spreadsheet modeling with Excel, Tiller°, and XML. As described in 
Chapter IV (Model Development) the author maintains that spreadsheet 
modeling provides an organized method to perform the thought process, while 


simultaneously cataloging important modeling parameters. 


1. Spreadsheet Modeling with Excel 


Appendix B, Section B, “DOE Spreadsheet Modeling,” outlines the 
crossed FRLH DOEs, both large (1,008 design points) and _ small 
(144 design points). There are six spreadsheets. 


1. The first is the factor description and is similar to that of 
Tables 17-19. It outlines both the decision’ (associated with first 
responders) and noise* (associated with civilians and terrorists) 
factors creating the robust design. 


2: The second spreadsheet is a FRLH coded spreadsheet for 
40 factors detailing the factor levels used at each of the 
1,008 design points in the large FRLH experiment.*? 


3. The third spreadsheet is a design file and looks very similar to the 
second. This file adds an additional 11 correlated factors. These 
are correlated to the civilian instance factors. The correlation 
represents an assumption that the author made in scaling all 
different classes of civilians to maintain the same proportionality as 
in the EPiCS simulation conducted by TRAC-WSMR._ The design 
file incorporates the final crossed NOLH DOE with 
3,024 design points. 


4. The fourth spreadsheet is a FRLH-coded spreadsheet for 
40 factors detailing the factor levels used at each of the 144 design 
points in the small FRLH experiment.” 


"Decision Variable: Variable that can be controlled by a decision maker for first-response 
forces. 


*’Noise Variable: Variable that cannot be affected by first response decision makers. 

°SE-mail from LTC Alejandro Hernandez titled “Roginski Designs,” sent 11 May 2006, office 
communication. FRLH 40 Factors, coded by LTC Alejandro Hernandez, Naval Postgraduate 
School, Monterey, CA. 


Ibid. 
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5. The fifth spreadsheet is a design file and looks very similar to the 
second. This file adds an additional 11 correlated factors. These 
are correlated to the civilian instance factors. The correlation 
represents an assumption that the author made in scaling all 
different classes of civilians to maintain the same proportionality as 
in the EPiCS simulation conducted by TRAC-WSMR. _ The design 
file incorporates the final crossed NOLH DOE with 
432 design points. 


6. The sixth spreadsheet results from Visual Basic code written by 
Major Chris Michel, USMC, that takes row data from the FRLH 
spreadsheets and put it in a format to be used later in the Tiller®. 
The author will describe this code in Section V.C.1. 


2. Extensible Markup Language (XML) 


Though Pythagoras offers an easily viewed GUI to input data values, 
analysts may also build Pythagoras scenarios and edit them using the XML, as 
all Pythagoras databases are stored and transmitted in XML. XML offers a 
simple and very flexible text format device derived from SGML 
(ISO 8879). Technicians originally designed SGML to meet the challenges of 
large-scale electronic publishing; XML also plays an increasingly important role in 
the exchange of a wide variety of data on the Internet.* Storing scenarios in 
XML permits the analyst to transmit scenario files quite rapidly over the Internet 
to perform data farming techniques. This process occurs with agencies such as 
the MHPCC and enables thousands of design points to run over a networked 


cluster of computers in a short amount of time.” 


3. Tiller® 


The Tiller, Version 0.7.0.0, Copyright 2004 Referentia Systems 
Incorporated, is a product developed in support of Project Albert and the 
Marine Corps Warfighting Laboratory. Its primary purpose is to prepare model 


°W3C, Extensible Markup Language. Retrieved on 21 May 2006 from the World Wide Web 
at http:/Awww.w3.org/XML 


°*°Charles A. Sulewski, “An Exploration of Unmanned Aerial Vehicles in the Army’s Future 
Combat Systems Family of Systems,” Masters Thesis, Naval Postgraduate School, Monterey, 
CA, December 2005. 
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XML scenarios for data farming. The Tiller® outputs two files the user must 


submit together to execute a data farming job. 


The first file is basecase.xml, which is the model scenario file. For 
Pythagoras models, the only change to the scenario file is in changing the name 
to basecase. The author identified that it can be more efficient to manually 
change the name of the scenario file, rather than use the file generated by the 
Tiller°. The file generated by the Tiller° can be up to 33% larger than the base 
scenario file, with no added functionality. The second file generated by the 
Tiller® is a usable study.xml file containing the chosen DOE for running at 
the MHPCC. 


The Tiller® application may be used alone to process the DOE. The 
author used the Tiller’s~ study.xmI file as a starting point from which to “lockstep” 
several values together. Lockstepping is the process by which several factors 
are changed, but some values are changed together. The result of submitting 
the altered study file was that the author changed 104 variables in each design 


point, using 37 different factors from Hernandez’s FRLH. 


4. Excel Interface by Michel 


Major Christopher Michel, USMC, developed Visual Basic code that 
provides a bridge between the FRLH developed by Hernandez and the Tiller® 
developed by Referentia. The user copies XML code that defines each 
excursion to be run. Michel’s code causes Excel to snap to the worksheet with 
the FRLH and copy a row of factors. Next, the program snaps to the worksheet 
that the user has prepared to receive the design points and transposes the 
factors into the appropriate cells. This process is repeated until the final row of 
the FRLH is placed into the spreadsheet. All that remains for the user to do is 


copy the first three columns of the worksheet and paste those cells into the 
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study.xmI file.°’ Refer to Appendix B, Section B (Excel Macro by Michel) to see 
the Visual Basic code. 

This chapter outlined the procedures for completing steps 11 and 12 of the 
simulation methodology discussed in Figure 5, which discusses the efficient 
experimental design used in this research. The following chapter will 
demonstrate the accomplishment of step 13, the analysis of the data resultant 


from implementing the DOE. 


“Christopher Michel, “Supporting A Marine Air Ground Task Force With Appropriate 
Quantities of Ground Based Fire Support,’ Masters Thesis, Naval Postgraduate School, 
Monterey, CA, September 2006. 
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VI. DATA ANALYSIS 


This chapter describes step 13 of the simulation methodology depicted in 
Figure 6, the analysis of data developed from executing the experimental design 
discussed in the previous chapter. It also addresses the tools and techniques 
used to collect and “clean” the data. Next, the thesis-based observations provide 
the analysis that answers our research questions. The insights into these 
research questions are summarized in the conclusions and recommendations for 


future study that comprise Chapter VII. 


A. DATA COLLECTION AND CLEANING 
1. Clementine 


Clementine is a data mining software package that contains features used 
throughout most of the data cleaning process. Data cleaning is the process by 
which raw data from an experiment are transformed into a format and layout that 
is conducive to conducting analysis. To facilitate this process, and other data 
mining processes, Clementine provides a GUl-based suite that enables quick 
and easy manipulation of the initial data sets into manageable sets for analysis. 

The feature of Clementine that was most important in data cleaning was 
the development of a data “stream” on the background palette.** A Clementine 
stream is an intuitive, graphical representation of a process on the computer 
screen. For example, this data cleaning process consisted of importing a data 
file, removing columns, changing column names to make data analysis more 
intuitive, and exporting the transformed data into another file. To repeat the 
process for another file, all that need happen is for the new file to be uploaded 
onto the palette, and then the stream attached to the new file. The new stream is 
executed and immediately cleaned. See Appendix C (Data Analysis) for an 
illustration of a Clementine stream used in this data cleaning process. 


°*8Clementine product description. Retrieved on 30 May 2006 from the World Wide Web at 
http:/Awww.spss.com/clementine 
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After all files from a given experiment were ready, the author imported the 
new files into Excel and inserted a column that identified the associated 
probability of communication for the given file. The new file was reloaded to the 
Clementine palette and appended to the other files that resulted from the 
experiment. This consolidated file was then exported into a file format to be 
analyzed with JMP Statistical Discovery Software™. The next section provides 
more information about JMP. 

Clementine facilitated the data cleaning process for ten total files, in three 
separate experiments, resulting in the removal of nearly 52,000 columns of data 
from the files to be analyzed. In Clementine, this process took approximately 
three hours. Based on preliminary work in SPLUS, executing the same process 
in SPLUS would have taken twice as long, even with its new Big Data library 
feature. In JMP, the process would have consisted of 52,000 point and click 


deletions, taking much longer than even SPLUS. 


2. Analysis Software Tools (JMP Statistical Discovery Software ™) 


JMP Statistical Discovery Software™ contains the software features used 
for both the data cleaning and the data analysis portion of this research. The 
JMP statistics package is the tool chosen to support the majority of the data 
analysis because it is a useful statistical analysis tool with powerful data 
visualization features. JMP excels at helping analysts uncover relationships and 
outliers within the data. The exploration of these data points can lead to valuable 
discoveries and_ possible surprises in the data, thus supporting 
better decision-making.” 


a Analysis Techniques 


Most large databases yield the flexibility to perform a wide array of data 
analysis techniques. Though this analysis applies statistical tests, the core 


S9uMP, The Statistical Discovery Software. Retrieved on 20 April 2006 from the World Wide Web 
at http:/Awww.jmp.com/product/jmp5_brochure.pdf 
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analysis focuses primarily on two techniques: Classification and Regression 
Trees, and Multiple Regression. 


a. Classification and Regression Trees (CART) 


The CART (Classification and Regression Trees) algorithm is a 
widely used statistical procedure for producing classification and regression 
models with a tree-based structure. Tree models assist the analyst in the 
identification of significant factors, by grouping similar points into cells. This 
process is recursive; it repeats as many times as necessary so that each end 
branch defines a separate node.” °' The regression tree yields a continuous 
output; however, classification trees are discrete in nature.” The CART 
algorithm will classify significant response factors into classes complemented by 


further regression analysis.” 


Throughout this analysis, the author used a 5% heuristic to 
determine when to stop splitting the tree. As additional splits are made to the 
tree, the R® value is logged. Each R* value is compared to the previous value, 
when the change in R? is less than 5%, no further splits are made. The final split 
is then pruned, leaving only splits in the tree that make an impact on the R® of 
greater than 5%. 


b. Multiple Regression 


A general regression analysis is a_ statistical process that 
investigates the relationship between two or more variables (factors) related in a 
nondeterministic fashion. The objective in multiple regression is to build a 





“Douglas Montgomery, Elizabeth Peck, and Geoffrey Vining, Introduction to Linear 
Regression Analysis, Third Edition, (John Wiley and Sons, Inc., 2001), p. 516. 


®'David Hand, Heikki Mannila, and Padhraic Smyth, Principles of Data Mining, (MIT Press, 
Cambridge, MA, 2001), p. 145, 343. 


®Hand, p. 147. 
ibid 47. 
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probabilistic model that relates a dependent variable y to one or more 
independent or predictor variables. The actual y values in a sample differ from 
the predicted values. The errors or residuals, denoted by e, are the differences 
between the observed and predicted values, hopefully possessing a normal 
distribution with constant variances.” Regression analysis is practical for gaining 
insight on which predictor variables (design factors) have the greatest 
significance toward the success of the first response mission, as measured by 
the previously mentioned MOEs. Regression analysis is also useful in identifying 


interactions between input variables. 


B. RESEARCH QUESTION-BASED ANALYSIS 


Each data set developed through the experimental design process 
contributed to the analysis described in this section. Because of data output 
difficulties at MHPCC, only two-thirds of the data from the small FRLH 
experiment was available for analysis, corresponding to the 144 design point 


base design, with probability of communication set at 1.0 and 0.75. 


Before conducting analysis of the data using trees and regression models, 
the data were grouped by like design points. This model is stochastic and, as a 
result, each design point is replicated 15 times. Design points are grouped and 
desirable statistics taken (e.g., means of desired end of run MOEs). 


For example, the raw data set for the small FRLH experiment resulted in 
4,320 rows. The first MOE addresses the percentage of civilians injured after the 
end of the first hour. The data set is grouped by design point, and the mean 
number of civilians injured at the end of the first hour taken. The result of this 
approach is that the actual data set analyzed for a given problem is more 
manageable than when in its raw form. A data that is 4,320 rows by 
170 columns (after cleaning) may be reduced to data set of 288 rows by 


39 columns. 


4Devore, p. 587. 
Ibid 47. 
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1. MOE 1. Percentage of Civilians Injured After First Hour 
a. Flexible Random Latin Hypercube (FRLH) Experiment 


The first step of this analysis was to develop a classification and 
regression tree to identify which variables have the greatest significance in 
determining the percentage of civilians injured. Figure 24 shows the increase in 
R? (a measure of the amount of variance in the data that is explained by the 
given model) with the number of splits in the regression tree, using data 
generated with the small FRLH model. Note that after the third split in the tree, 
there is only a small difference in R®. The increase resulting from making four 
splits, versus three, is only 0.03, from 0.816 to 0.846, resulting in a well-fit 
regression tree. 
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Figure 24. Analysis of R” by Regression Tree Split (Small FRLH Experiment, 
Response = Percentage of Injured Civilians) 


Figure 25 is the regression tree developed in JMP to help gain 
insight into the factors that influence the percentage of civilians that remain 
injured after the first hour. As mentioned previously, regression trees involve the 


use of an algorithm that groups like observations together, maintaining the 
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highest possible “purity.” After the completion of each step, the algorithm 
calculates the decrease in total sum of squares resulting from the introduction of 
each “candidate node.” See Appendix C (Data Analysis) for a more 


detailed description. 
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Figure 25. Regression Tree to Determine the Percentage of Civilians Injured After the 
First Hour (Small FRLH Experiment) 


A strength of regression trees is their interpretability and ease of 
understanding. For example, see item one on Figure 25. Item one is the first 
split of the regression tree. JMP determined that the division in the data that 
would result in the greatest increase in purity would be to split the data using the 
variable “Police Effectiveness in Issuing Orders to Civilians.” In effect, this is the 
most important single variable in determining the number of injured civilians at 
the end of the first hour. The increase in purity corresponds to the greatest 
reduction in total sum of squares. The tree is used as follows. 

e There are 64 design points that correspond to a _ police 
effectiveness score of less than 23%. For these cases, an average 


of about 15% of the civilians remain injured at the end of the first 
hour (see the boxed entry in the first split of the tree). 
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e There are 224 design points that have a police effectiveness score 
of greater than or equal to 23%. For these cases, an average of 
approximately 5% of the civilians remains injured after one hour. 


Item 2 shows the second split in the tree, corresponding to the 
greatest reduction in total sum or squares for the observations in the node 
(Pol_Effect<.23). Notice that the worst case scenario in the tree results from this 
split. If the police effectiveness is less than 6%, we would expect to see an 
average of about 20% of the civilians that remain at the end of the first hour. 

Item 3 shows the third split in the tree, corresponding to the third 
largest reduction in total sum or squares for the model. Notice that the best case 
scenario in the tree results from this split. If police effectiveness is less than 
23%, and greater than 43%, we would expect to see only about approximately 
4% of the civilians injured at the end of the first hour. 

It is interesting that, after considering all 37 independent variables, 
the most important variable in determining the number of injured civilians is a 
variable that the decision maker can control; it is a decision variable. However, 
the variable that has the second most impact is a variable which the 
decision maker cannot control, a noise variable—the desire of civilians to flee 
from the enemy. This variable is not displayed on the tree; it is the second 
variable to appear if the tree is expanded. This desire may correspond to a 
feeling of terror in identifying the enemy that is greater than the comfort offered 
by a first responder. The injured civilian runs away and the first responder is not 
able to help. 

It is helpful to understand that there is a certain degree of 
randomness that prevails in our world. A decision maker will mostly likely not be 
able to remove all uncertainty from a crisis response plan. It is helpful to 
understand the relationship of controllable factors to uncontrollable factors. The 
contour plot in Figure 26 helps demonstrate the relationship of our decision factor 
(police effectiveness) to the noise factor (desire to flee). 


103 


Contour Plot for Percent_Injured 
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Figure 26. Relationship of Police Effectiveness in Giving Orders to Number of Civilians in 
Determining Percentage of Injured Civilians (Small FRLH Experiment) 
(best viewed in color) 


This picture helps to illustrate the robust nature and importance of 
police effectiveness in giving orders to the civilians in the crowd. Note that when 
police effectiveness is at just 60%, the number of injured civilians does not 
increase appreciably, even at the highest levels of civilian desire to run from the 
enemy. It is only when effectiveness drops below 20%-30% that the number of 
civilians injured begins to increase with a greater civilian desire to flee. Note that 
the first split of the regression tree is when the police order effectiveness 
was 23%. 


b. Gridded Experiment 


The analysis shows that the effectiveness in police issuing orders 
to civilians is a dominating factor in determining the percentage of civilians 


injured at the end of the first hour of response. There are other questions that 
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must be analyzed to gain an understanding of what is most important in 
emergency response. 
° What role does communication degradation § play in 
emergency response? 


e Is there key terrain that the emergency response teams must 
ensure is secured to facilitate response? 


The technique chosen to gain insight into those questions was to 
run a gridded experiment in which just three variables are changed: probability 
of communication, number of patrolmen at each location, and number of traffic 
police at each intersection. Probability of communication did not enter into the 
tree as a significant variable in the small FRLH experiment. In reality, 
communication is very important in emergency response, and one of the points 
highlighted first in most interagency AARs. The author was only able to vary 
communication over two levels in the small FRLH experiment. The gridded 
experiment resulted from varying communication and both other variables at five 
levels each. 

Figure 27 shows the increase in R®, with the number of splits in the 
regression tree, using data generated with the gridded model. Note that 
diminishing returns in R® occurs after the fifth split. The increase resulting from 
making five splits, versus four, is only 0.025, from 0.642 to 0.667. The result is a 
tree that has an R? of only 0.642, lower than we would like for predictive 
purposes; however, the purpose of this tree is not necessarily prediction, but 


gaining insight into the relationship of factors that the decision can control. 
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Figure 27. Analysis of R” by Regression Tree Split (Gridded Experiment, 
Response = Percentage of Injured Civilians) 


Figure 28 is the regression tree developed in JMP, from the gridded 
data set, to help gain insight into the factors that influence the number of civilians 
that remain injured after the first hour. Within the tree, there is very little 
variability between the tree’s best (approximately 91%) and worst case scenario 
(approximately 87%). This small amount of variability means that it may be 
difficult to fit a good model to the data. 
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Regression Tree to Determine Percentage of Civilians Injured After the First Hour (R2 = 0.642) 
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Figure 28. Regression Tree to Determine the Percentage of Civilians Injured After the 
First Hour (Gridded Experiment) 


After making five splits, it is evident that communication does not 
seem to have significant importance in this model’s emulation of first response. 
In fact, it is not until the tenth split (not shown) that the communication factor is 
included in the tree. 

In retrospect, the lack of communication importance in this model 
vignette is not terribly surprising. Communication is meant to serve many 
functions in response, including, but not limited to: passing situational 
awareness in the form of information, sending information about enemy locations 
and dispositions, and providing direction in terms of orders. 

As previously stated, in Section IV.A.5., the actions simulated in 
this research are the adaptation of work that occurs by standing operation 
procedure (SOP). SOPs are meant to minimize the need for communication and 
that 


communication factors may be dominated by others, such as location and 


streamline operations; therefore, it is intuitive in certain cases, 
numbers of forces, in the early stages of an emergency. However, a model could 


easily be adjusted to account for lack of SOPs, undisciplined following of SOPs, 
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or SOPs that cannot be followed because of the situation. In these cases, 


analysis may reveal a higher level of importance for communication factors. 


2. MOE 2. Percentage of Civilians at the Medical Triage Point 
After One Hour 


Analysis of the second MOE follows the same pattern as the first. 
Figure 29 shows the increase in R* with the number of splits in the regression 
tree. Note that after the second split in the tree, the increase in R’ is small. The 


increase resulting from making three splits, versus two, is only 0.042, from 
0.791 to 0.833. 
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Figure 29. Analysis of R” by Regression Tree Split (Small FRLH Experiment, 
Response = Civilians at Triage Site) 


Figure 30 shows the regression tree that results from splitting twice. 
Notice that police effectiveness in giving orders is the most important variable; in 
fact, it is the only variable involved in the tree for the first four splits. 
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Regression Tree to Determine 
Number of Civilians At Triage Point (R? = 0.791) 
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Figure 30. Regression Tree to Determine the Percentage of Civilians at the Medical Station 
(Small FRLH Experiment) 


In the regression tree, police effectiveness has a dramatic impact on the 
percentage of civilians at the triage point, accounting for nearly 88% of the 
variance after four splits. Figure 31 shows multiple regression output that serves 
to verify the relationship of police effectiveness to percentage of civilians at the 
triage point. The step history indicates that police effectiveness is the first 
variable to enter the model, explaining about 77% of the variance. In the effect 
test, this variable seems to have a dramatically higher impact on the model than 
any other variable, shown by the F-test of 1,250, versus 25 and 20 for the next 


two highest variables. 
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Step History 


arameter Action "SigProb' SeqSS RSquare 
Ox: Entered 0.0000 1.733161 0.7659 
Agitator_MarksmanshipEntered 0.0000 0.033254 0.7806 
NUM_Ag_4 Entered 0.0000 0.028562 0.7932 
Num_ Civilians Entered 0.0000 0.027852 0.8055 
Num_Traffic_8 Entered 0.0046 0.012378 0.8110 
Num_EMT_SJ Entered 0.0111 0.009729 0.8153 
Gunman_Vuln Entered 0.0271 0.007238 0.8185 
Num_Traffic_3 Entered 0.0313 0.006782 0.8215 
Num_Fol_1 Entered 0.0329 0.006574 0.8244 
Gunman_Marksmanshif&ntered 0.0356 0.006297 0.8272 
Num_Ag_2 Entered 0.0396 0.005964 0.8298 
Num_ESU_3 Entered 0.0380 0.00599 0.8325 
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Effect Tests 


Source DF Sum of Square: _ F Ratio 
Num_Ag_2 1 0.0061529 4.4632 
NUM_Ag_ 4 1 0.0280710 20.3620 
Num_EMT_SJ 0.0100637 7.3000 
Num_ESU_3 0.0059898 4.3449 
Num_Fol_1 0.0059475 4.3141 
Num_Traffic_3 0.0065470 4.7490 
Num_Traffic_8 0.0105662 7.6644 
Num_ Civilians 0.0290635 CO+0 
Pol_ Effect 1.72561 {251.719 
Gunman_Vuln 0.0073219 se 
Agitator_Marksmanship 0.0345315 25.0482 
Gunman_Marksmanship 0.0059173 4.2922 





Figure 31. Stepwise Regression Output from JMP, Response = Percentage of Civilians at 
Triage Point, Additive Model 


Figure 32 shows the details of the multiple regression model developed. 
The original model is the one located on the left-hand side of the figure, an 
additive (main effect) model. Notice the Residual by Predicted Plot; there 
appears to be a quadratic relationship between the effectiveness of police orders 
and percentage of civilians at the triage point. After adding the quadratic term for 
police effectiveness (on the right-hand side of the figure), notice that the plots are 
much more well formed, with an R? of approximately 90%. 
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e Point Actual 
e_ Point Actual 


g 
w 
g 





Percent_At_Tria 
i) 


Percent_At_Tria 


.30 40 
Percent_At_Triage_Point Predicted 


Percent_At_Triage_Point Predicted 
P<.0001 RSq=0.83 RMSE=0.0371 


P<.0001 RSq=0.90 RMSE=0.0289 


o 
=) 
3S 
wn 
® 
o 
a= 
= 
[o) 
ot 
© 


[_Trlag 
oO 
Oo 
ge_Point Residual 


Percent_At_Tria 


Percent_At_Tria 


10 .20 .30 40 
Percent_At_Triage_Point Predicted 


Percent_At_Triage_Point Predicted 








Figure 32. Stepwise Regression Plots 
3. MOE 3. Neutralization of Enemy Gunman 
a. Small FRLH 


Analysis of the third MOE follows the same pattern as the first two. 
There is one difference in the generation of this MOE. There are several 
simulation runs in which the police force does not kill all of the gunmen. To 
account for this result, the author generated a binary variable. A value of 1 
means at least one terrorist survives, while 0 means that all are dead. As the 
data are grouped and the means collected, the Os and 1s are averaged to obtain 
the proportion of those 15 replications for which all of the gunmen are killed. 

Figure 33 shows the increase in R® with the number of splits in the 


regression tree. Note that after the fourth split in the tree, there is no appreciable 
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difference in R*. The increase resulting from making four splits, versus five, 
only 0.036, from 0.569 to 0.605. 
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Figure 33. Analysis of R” by Regression Tree Split (Small FRLH Experiment, 
Response = Percentage of Gunmen Neutralizations) 


Figure 34 shows the regression tree that results from splitting four 
times. Police effectiveness has importance in this model, but it does not show up 
as a factor until the third level, or fourth split. Of the four variables included in 


this model, only one is a factor that a decision maker on the first responder side 
can readily affect. 
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Regression Tree on Small FRLH Data to Determine Effectiveness 


in Neutralizing the Gunman (R? = 0.57) 





Count 288 
Mean 0.8310185 
Std Dev 0.1924495 























Num_Gunman>=4 


Count 144 
Mean 0.7412037 
Std Dev 0.2098575 


























| Agitator_Vuln<0.06 





Count 6 
Mean 0.1555556 
StdDev 0.116746 























| Agitator_Vuln>=0.06 





Count 138 
Mean 0.7666667 
Std Dev 0.1726188 
































Num_Gunman<4 


Count 144 
Mean 0.9208333 
Std Dev 0.1185168 

















Police_Order_Marksmanship<0.45 


Police_Order_Marksmanship>=0.45 





Count 64 
Mean 0.6760417 
Std Dev 0.1716677 








Count 74 
Mean 0.845045 
Std Dev 0.1306153 


























| Agitator_Vuln<0.48 


| Agitator_Vuln>=0.48 





Count 
Mean 


StdDev 0.168473 


30 Count 34 
0.5733333 Mean 0.7666667 
Std Dev 0.1151781 














Figure 34. Regression Tree on Small FRLH Data to Determine Percentage of 
Gunmen Neutralized 


Number 1 in Figure 34 shows the best case scenario: if there are 


less than four gunmen, we expect all of the gunmen to be neutralized 


approximately 90% of the time. 
Number 2 in Figure 34 illustrates the worst case scenario. Below is 


the interpretation of this leaf in the decision tree. If 


there are more than four gunman, and 


agitator vulnerability is greater than 6%, and 


police marksmanship is less than 0.45, and 


if agitator vulnerability is less than 48%, 


then you would expect all terrorist gunmen to be killed only about 


half the time. 
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Agitator vulnerability is involved in two of the four levels of this tree, 
highlighting a possible secondary effect that is intuitive. If the agitators are less 
vulnerable, they draw the attention of the first responders away from the gunmen. 


b. Large FLRH Experiment 


One of the strengths of MAS is the ability to “grow” data to analyze 
a given problem in greater and greater levels of detail. Analysis of the 
effectiveness in neutralizing the terrorist gunman is an example of this strength. 
The small FLRH experiment yielded several instances in which all of the 
terrorist gunmen were not killed. To ensure accurate insight is gained into the 
effectiveness of neutralizing the gunmen, more data can be “grown” using the 
principles of data farming. In this case, the large FRLH experiment resulted in a 
data set that was more than 10 times the size of the small experiment 
(45,360 rows versus 4,320 rows), enabling analysis at a finer level of detail. 

Figure 35 shows the increase in R* with the number of splits in the 
regression tree. Diminishing returns in R* occurs after the fifth split. The 
increase resulting from making six splits, versus five, is only 0.047, from 
0.534 to 0.581. 
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Figure 35. Analysis of R” by Regression Tree Split (Large FRLH Experiment, 
Response = Percentage of Gunman Neutralizations) 
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The regression trees resultant from modeling the large and the 
small FLRH have a key difference. In the large FRLH experiment, police order 
effectiveness is the most important factor; the single variable that explains the 
most variance (see Figure 36). 
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Figure 36. Regression Tree on Large FRLH Data to Determine Percentage of 
Gunmen Neutralized 


115 


As illustrated in Figure 36, only two factors (of a possible 38) are 
unique in the first six splits. See Figure 37 for a contour plot that highlights the 
relationship between police order effectiveness and the number of 
terrorist gunmen in determining if all the gunmen will be killed. The coarseness 
of the plot is explainable in part by the variables representing the 36 dimensions 
not captured on the axes. The picture contains an overlay of the splits made on 
the regression tree. 
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Figure 37. Relationship of Police Effectiveness in Giving Orders to the Number of 
Terrorist Gunmen in Determining Percentage of Times Gunmen Killed 
(Large FRLH Experiment) 

(best viewed in color) 


The relationship between the number of gunmen in the simulation 
and the difficulty in killing them all is obvious. It may not be obvious why the 
effectiveness in the police officers’ use of a paintball weapon impacts the survival 
of the gunmen. After viewing the simulation several times, a possible 
explanation lies in the police officers’ focus. Simply put, they do not do two 
things at once very well. As the police order effectiveness increases, more 
civilians move to the triage points faster, resulting in the police having the ability 
to focus on killing the gunmen. Based on the settings given to the police, it is 
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expected that, when more civilians are in the area, the police will be distracted by 


the additional civilians and not as responsive to the gunmen. 


4. MOE 4. First Responders Killed or Wounded 


Figure 38 shows the increase in R® with the number of splits in the 
regression tree. Note that after the third split in the tree, there is no appreciable 
difference in R*. The increase resulting from making four splits, versus five, is 
only 0.035, from 0.687 to 0.722. 
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Figure 38. Analysis of R” by Regression Tree Split (Small FRLH Experiment, 
Response = Proportion of First Responders Killed or Injured) 


Figure 39 shows the regression tree that results from splitting three times. 
Police effectiveness is again the most important factor in predicting our response 
variable, the proportion of first responders killed or injured in the first hour. 
Figure 40 shows a check done with linear regression to ensure that similar 
variables are included with an additive multiple regression model (without 
interactions). Fortunately, both methods yield similar results, both in variable 
selection and in R’; therefore, we can be confident that we are truly capturing the 


most important variables in the scenario. 
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Regression Tree on Small FRLH Data to Determine Proportion of 
First Responders Killed or Injured (R2 = 0.69) 
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Figure 39. Regression Tree on Small FRLH Data to Determine Proportion of 
First Responders Killed or Injured 
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| Step History 





Step Parameter Action "SigProb" SeqSS RSquare Cp p 
1 Police_Order_Marksmanship Entered 0.0000 4.707798 0.4733 385.61 2 
2 Agitator_Yuln Entered 0.0000 0.931277 0.5669 268.59 3 
3. Agitator_Marksmanship Entered 0.0000 0471817 O6143 2105 4 
4 Num_Gunman Entered 0.0000 0.338301 0.6483 169.06 5 
5 Num_Ag_2 Entered 0.0000 0.206709 0.6691 144.65 6 
6  Num_EMT_SJ Entered 0.0003 0.153511 0.6845 127.03 7 





Figure 40. Results of Multiple Regression (Stepwise) Variable Selection (Small FRLH 
Experiment, Response = Proportion of First Responders Killed or Injured) 
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It is interesting that police marksmanship, or effectiveness in giving orders 
to civilians, has a dramatic impact on the proportion of first responders that are 
killed or injured. This type of “marksmanship” is associated with a paintball 
weapon that should not directly affect first responders. Further inspection of the 
scenario and the model will determine whether police order marksmanship has 
its effect directly, or through a synergistic effect or emergent behavior caused in 
other agents. 

Figure 41 illustrates the relationship between vulnerability of terrorist 
agitators and effectiveness of police in using their paintball “orders” weapon. The 
data points are included in this plot to give the reader an idea of the data filling 
nature of Hernandez’s FRLH design. 
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Figure 41. Relationship of Police Effectiveness in Giving Orders to Terrorist Agitator 
Vulnerability in Determining Proportion of First Responders Killed or Injured 
(Small FRLH Experiment) 


The relationship depicted in Figure 41 is as one would expect, if the 
weapon addressed with police marksmanship is a lethal weapon. Because it is a 
paintball weapon, it is important to check the model settings and ensure that 
there is no artifact in the simulation that is causing the weapon to have a 
capability that the weapon should not have. 
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This chapter was an analysis of the data that resulted from execution of 
the simulation model developed in accordance with steps 9 and 10 of the 
methodology established in Figure 5 of this work, and run using efficient 
experimental design, steps 11 and 12. Given more time, much more analysis 
could be accomplished with this data. The data analysis began with gaining 
quick insight into the data, using a small FRLH design. Areas of particular 
interest were brought into further focus using two more detailed approaches, a 
gridded design, and a larger, more space filling Latin Hypercube design. The 
final chapter of this research provides the conclusions gained from the 
development of the first response simulation methodology and analysis of the 
simulation data. In addition to the conclusions of this study, Chapter VII includes 


recommendations for future study. 
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Vil. CONCLUSIONS AND RECOMMENDATIONS 
FOR FUTURE STUDY 


This chapter summarizes the contributions of this research, describing 
both general methodological contributions and specific findings that relate to the 
established research questions. Following the discussion of conclusions, the 
author provides recommendations to government agencies for planning and 
coordinating emergency first response. Recommendations for future study 
comprise the final section of this research, outlining ways in which this work can 
be carried forward. 


A. CONCLUSIONS AND CONTRIBUTIONS 
1. Methodology Contributions 
a. Organizational Learning Methodology 


Organizational learning is a process by which an organization 
deliberately expands its knowledge base for a certain purpose. The methodology 
proposed in Figure 2 of Chapter | illustrates how to use low resolution simulation 
in conjunction with high resolution simulation to facilitate the organizational 
learning process during preparation for a TOPOFF exercise. Application of this 
methodology can certainly result in more effective planning, a more focused 
exercise, and a better-trained response force. In addition, application of this 
methodology could result in more efficient use of sparse resources, such as 


training dollars and manpower. 


b. Emergency First Response MAS Methodology 


Simulation of emergency first response provides decision makers 
with a tool that provides insights that cannot practically be gained through other 
means. The ability to analyze tens of thousands of different situations and 
variable combinations provides the analyst with the ability to gain a much more 
thorough knowledge of a problem’s response surface than can be provided by 
subject matter expertise or point estimates, gained through high resolution 
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simulation. The methodology presented in Figure 6 of Chapter Ill shows a 
step-by-step process that an analyst can follow to use a low resolution simulation 
to credibly analyze emergency first response to a crisis situation. 


Cc. Experimental Design Methodology 


Efficient experimental design is the bridge that enables the MAS to 
be analyzed in several dimensions, fully leveraging the power of modern 
computers. The experimental design methodology developed in Chapter V 
provides the analyst with a mechanism to take a broad approach to initial 
analysis, using Hernandez’s FRLH designs. The analyst then narrows the focus 
on areas of interest, using a space filling NOLH design, such as those developed 
by Cioppa. Finally, to isolate the effects of a small number of variables, the 
analyst can utilize a gridded design. 


2. Analytical Conclusions 


The analytical results here demonstrate a proof of concept. It is possible 
to gain useful insight through the utilization of a low resolution model, such as a 
MAS. Although it is possible to provide decision support analysis, the findings in 
this research should not be directly applied to the TOPOFF process or to 
Baltimore first responders. To provide analysis that is relevant to a given 
agency, it is imperative that accurate local force structure and SOPs be obtained. 
In addition, local subject matter experts (SMEs) should be involved in model 
verification. When these steps are taken, it is possible that insights such as 
those described in Chapter 6 can be relevant to first response organizations. 

Overall, the most important factor in determining the effectiveness of 
first response forces is the effectiveness with which the police forces are able to 
calm the civilians and direct them to the triage points. Due to the ROE used for 
the police in this simulation, they fixate on the civilian they are assisting, until that 
civilian receives the help he or she needs. If the police are effective, they can 


then move on to other priorities of work, such as addressing an injured unit 
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member or terrorist threat. If the police are not effective, they do not disengage 


and execute another mission. 


a. What is the Most Appropriate Mix of Response Forces? 


This analysis provided insight into the determination of response 
force mix by focusing on two fundamental first response MOEs: 


e Percentage of civilians not treated in the first hour 


° Percentage of civilians relieved of panic in the first hour 


Although not able to determine a true recommended mix of police 
forces to medics to firemen, the analysis showed that the number of responders 
is not as important as their effectiveness. 

(1) Number of Civilians Injured After the First Hour. This 
MOE measures the effectiveness of the first responders in addressing the 
civilians’ physical needs, i.e., first aid and medical stabilization. Effective police 
interpersonal relationship skills with the crowd is the most important factor in 
determining how many civilians are injured after the “Golden Hour,” the first hour 
after an incident. At medium and high levels of police effectiveness, only a few 
injured civilians are able to depart the first response area of influence without 
being given first aid by a policeman or treated by a medic. It is only at very low 
levels of effectiveness that the police become “overwhelmed” by the number of 
civilians in the area of the bomb blast and are not as effective. 

(2) Percentage of Panic-Relieved Civilians After the First 
Hour. This MOE measures the first response force’s ability to address the 
civilians’ psychological needs, i.e., the ability to calm the crowd and restore 
order. Police effectiveness in maintaining crowd control is again the most 
important factor in addressing the needs of civilians, and clearly dominates all of 
the other 36 factors considered in the analysis. 
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b. What is the Impact of Communication Effectiveness in 
Emergency First Response to a VBIED in Baltimore’s 
Inner Harbor? 


Communication effectiveness plays a key role in the ability of 
command and control elements to relay information vertically and laterally to 
increase response force situational awareness in the response area. This 
simulation analyzes possible physical characteristics of communication, 
i.e., effectiveness of a given piece of equipment. This model does not attempt to 
simulate the human dimension of communication, such as measuring the 
equivalence of the message sent versus the message received. 

(1) Neutralization of Terrorist Gunmen. In this simulation, a 
key feature is the passing of information from the first responders to an incident 
command cell. The incident command cell then processes and laterally passes 
the information to other command cell members; the consolidated information is 
disseminated to unit members. One such piece of information is the location of 
terrorist gunmen. The analysis showed that communication was not a key factor 
in the neutralization of the enemy gunmen; rather, police effectiveness in giving 
orders to civilians and the number of gunmen were the dominant factors. 

A possible explanation exists for the counterintuitive 
importance of police orders. Importance of orders may lie in the ROE. In this 
simulation, police have a higher desire to help the civilians than kill the enemy. 
As such, with decreased police effectiveness, the civilians are not helped as 
quickly. The police, engaged in trying to assist the civilians with decreased 
effectiveness, do not disengage from the civilians in order to address the threat 
of gunmen. 

The lack of importance of communication may result from 
the procedural nature of first response. Effective first response involves effective 
execution of SOPs. Communication, while important to maintain situational 
awareness, may not be a key factor in first response. Agencies know what to do 
when they arrive on scene, most likely have trained on it, and are prepared to 


execute in accordance with local SOPs. Communication becomes of paramount 
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importance in the execution of the follow-on mission, as police, fire, and medical 
forces arrive from outside the crisis area (not modeled in this simulation). The 
incident command post must dispatch these units to the appropriate area, 
requiring communication and situational awareness. This simulation ends before 
these additional follow-on forces arrive, a possible reason why communication 
does not appear as a relevant factor in this analysis. 

(2) Proportion of First Responders Killed or Wounded. The 
proportion of first responders killed or injured relates to the importance of 
communication in a similar manner to the neutralization of the terrorist gunmen, 
but adds an additional dimension. Police must neutralize the lethal threat of the 
gunmen, or incur additional casualties. All first responders must be able to 
communicate their injury status to unit members to be able to receive 
medical attention. 

Communication does not appear to be a relevant factor in 
determining the proportion of first responders killed or injured. Rather, the most 
important factor is police effectiveness in addressing civilian needs. As 
suggested in section VI.B.1.b, this may have been the result of modeling SOPs. 
Again, a model could easily be adjusted to account for lack of SOPs, 
undisciplined following of SOPs, or SOPs that cannot be followed because of the 
situation. In these cases, analysis may reveal a higher level of importance for 


communication factors. 


B. RECOMMENDATIONS FOR EMERGENCY PREPAREDNESS 


Simulation is a valuable tool that first response agencies could use to 


facilitate emergency preparedness in two ways: 
e Training of leaders and staff 
° Exercise planning 
1. Training of Leaders and Staff 


Emergency response organizations at virtually all levels should use 
simulation as part of their training program for leaders and staff. Planning, 
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executing, and analyzing a simulation provides a forum that helps leaders and 
staff better understand local SOPs. Executing a credible simulation requires the 
simulation to emulate reality. To prepare the simulation, the modeler must 
thoroughly understand what the SOP says, but also what it means: the effect 
meant to be achieved by a certain directive. During the execution of the 
simulation, the training audience may see emergent behavior by agents that 
causes them to question and improve certain facets of the SOP. During the 
analysis of the data generated by the simulation, the leaders and staff may 
encounter surprises in the data that also causes the reexamination of portions of 
the SOP. 


2. Exercise Planning 


Planners can make large-scale exercises more effective training tools 
through the use of the organizational learning process that incorporates the MAS 
modeling methodology established by this research. Through the development 
and execution of simulations, leaders and staff develop an understanding of the 
given problem that is simply not gained through less structured brainstorming 


and planning sessions. 


C. RECOMMENDATIONS FOR FUTURE STUDY 


Analysis of a given problem can result in the development of as many or 
more questions than it solves. This analysis is no exception. The author used a 
very specific set of assumptions to complete this simulation. The simulation and 
the subsequent analysis can be considered one data point. Many more data 
points are needed to better understand the full capability represented in 
this research. 

This research included a cursory qualitative comparison of the MAS 
developed to a well-established, high resolution model, EPiCS. A future project 
could be the development of a methodology to quantitatively assess the 
calibration of this model to EPiCS. The work could then generalize the 
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comparison methodology to include the comparison of other MAS simulations to 
appropriate high resolution simulations. 

In addition to being calibrated by another model, this model can be 
calibrated using actual events. With the assistance of the DHS or the 
City of New London, this model could be populated with actual force structure 
and SOPs associated with the response forces for TOPOFF 3. The model could 
then be run and analyzed to determine how close it came to replicating the actual 
events of TOPOFF 3 in New London. 

Typically in crisis situations, the affected civilians react in different ways. 
Future work could involve the representation of civilians that assist emergency 
responders in the accomplishment of their mission. In addition, it is possible to 
model civilians that work against first response forces, e.g., by looting or even 
attacking first responders. Interaction with helpful civilians or antagonistic 
civilians may sway a neutral civilian to one side or the other. The author led 
preliminary work in this area during the 12" Project Albert International Workshop 
(PAIW), held in Boppard, Germany. See Appendix D (PAIW — 12 Findings) for 
details. 

This simulation involved the execution of one set of first response and 
enemy courses of action. Follow-on work could examine the execution of 
different courses of action for both first response and terrorists. It would be 
interesting to note first response courses of action that are most robust against a 
variety of terrorist courses of action. 

Another possible project that could spring from this work is analyzing the 
usefulness of separate vignettes from the model in answering research 
questions. It is possible that some research questions do not require the 
aggregation of all agents and agencies. Disaggregation of the model could result 
in much lower processing times for simulation runs, meaning a quicker turn 
around of data for analysis. While losing the synergistic effects of including all 
agencies in the model, disaggregation may be- appropriate in 


some circumstances. 
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The Pythagoras scenario files, Excel modeling files, and all cited 
electronic references are available by contacting MAJ Jonathan Roginski, 
United States Military Academy, Department of Mathematical Sciences, 


West Point, New York, or by e-mail at jonathan.roginski@us.army.mil. 
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APPENDIX A — MODEL IMPLEMENTATION 


This appendix provides detailed information about the numbers the author 
used in this research. Included are details about how distances were measured 
for the scale of the simulation, terrain settings, weapon characteristics, and side 
affiliations of agent classes. The purpose of this appendix is to make this 
simulation as transparent as possible, to reveal what is going on 


“under the hood.” 
A. OVERALL MODEL CONFIGURATION 


1. Scaling 


Figure 42 is a terrain snapshot of the Celebrate Baltimore area, taken 


using Google Earth, centered on the Amphitheater in Baltimore’s Inner Harbor. 


TWh eis cetst ' | Waters ii 
Ls ' 4 





Figure 42: Terrain Snapshot of Celebrate Baltimore Area® 


Image downloaded from the World Wide Web using Google Earth on 12 April 2006. 
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To measure the actual size of this representation, the author used the 
latitude and longitude measurements at the four corners of the picture and 
entered them into a latitude/longitude to feet distance converter found at: 


http://www.epa.gov/athens/learn2model/part-two/onsite/Il-dms.htm. 


See Figure 43 for the width of the above snapshot and Figure 44 for 
the height. 


EPA On-line Tools for Site Assessment Calculation 
Recent Additions | Contact Us | Print Version iGO] 


EPA Home > > Ecosystems Research > Modeling Subsurface Petroleum Hydrocarbon Transport > OnSite on-line calculaton 








Lat-Long (degrees-minutes-seconds) to distance 


Module Home Objectives Table of Contents Previous < Next > 








Distance from Latitude and Longitudes (degrees-minutes-seconds) 





Example Data | Calculate Clear 


Location Name Latitude Longitude 

pe he fi ee 

[| bs_ fir] se _ fe ee] 
feseaa7 feet 


Same calculation in decimal latitude and longitude. 





Note: This calculation is not accurate enough for navigation 


The calculations assume the Earth is a perfect sphere (which it is not). These 
results are intended for use with GPS receivers used to locate features on 
leaking underground storage tank sites. Check the accuracy of your GPS 
receiver as large errors occur when used for small scale measurements. 


Go Back 

















Figure 43: Calculation of Terrain Box Width 





Lat-Long (degrees-minutes-seconds) to distance 


Module Home Objectives Table of Contents Previous < Next > 


Distance from Latitude and Longitudes (degrees-minutes-seconds) 
Example Data | Calculate Clear | 


Location Name Latitude Longitude 


fo fie fe fee ar 
a A 2 Ce 
fea3999 feet ha 


Same calculation in decimal latitude and longitude. 











Note: This calculation is not accurate enough for navigation. 


The calculations assure the Earth is a perfect sphere (which it is not). These 
results are intended for use with GPS receivers used to locate features on 
leaking underground storage tank sites. Check the accuracy of your GPS 
receiver as large errors occur when used for small scale measurements. 


Go Back | 

















Figure 44: Calculation of Terrain Box Height 
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The goal of achieving a 2,000-foot by 2,000-foot square box would be 
achieved by making a smaller box that is approximately 67% of the current box’s 
width and approximately 80% of its height. 


2. Terrain 


Figure 45 illustrates the selection concealment factor and selection of 
ceiling height. The modeler decides on a concealment factor that guards against 
detection in the three sensing spectra. These factors are located in the GUI ona 
sliding scale, from 0.0 to 1.0 (least visible to most visible). Movement and 
protection factor are located below concealment, and are scaled in the same 
manner as concealment. A movement factor of 0.0 means that an agent cannot 
move within this terrain feature, while a factor of 1.0 results in unencumbered 
movement. A protection factor of 1.0 results in total protection, while 0.0 results 


in no protection. 


eature List 
Unnamed Street 6 (Diagonal) 
Building 4 
Building 1 Inside 











[Building 4 | Last Modified: [today 











Description: {Large building directly south of amphitheater 


Feature Properties 


Concealment 





Choose Color 





Concealment # 








0.0 0.1 02 03 04 05 06 07 08 0.0 0.1 02 03 04 05 06 07 08 09 10 00 01 O02 03 04 05 06 07 08 O9 1.0 


Movement Factor: 0.0 Protection: 1.0 


0.0 0.1 02 03 04 05 06 07 08 09 10 00 01 02 03,04 0.5 OWS. 











ma eee Ceiling Height 
= Floor Height: 0 ‘Ceiling Height: 28 (in Pixels) 








0 100 200 300 400 500 600 700 800 9001000 0 100 200 300 400 500 600 700 800 9001000 





Figure 45: Instantiating Terrain Factors 


Figure 46 shows the Pythagoras representation of the terrain located in 
Figure 42. 
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| Building Patapsco River and 
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Figure 46. Pythagoras Representation of Celebrate Baltimore Area 





5 Weapons 


Figures 47-49 show the weapon settings used in this simulation. In 
Figure 47, fire rate per minute and engagement range each have columns that 
represent the actual scale values and their corresponding values as scaled for 
this Pythagoras simulation. The workbook used to track weapon characteristics 


was adapted from research completed by Major Michael Babilot, USMC.* 


°’Michael Babilot, “Comparison of a Distributed Operations Force to a Traditional Force in 
Urban Combat,” Masters Thesis, Naval Postgraduate School, Monterey, CA, September 2005. 
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——feneral Weapon Characteristics 


Max |Pythag| —s'|[Random| si 
[$$ $e Effect- — — Engage |Max Basic |Damage Suppress] | — 
pee liveness per ull FireRate | Range(m) {Range load meaice. Duration erst Fire 
M4 





Agitator 

Bomb (Carleton) 

Bomb (Cookie Cutter) 
Medical Kit 

Orders - From Firemen 
Orders - From Medics 
Orders - From Police 





ooooocoo°co]°o 
<<<<ZzZz2<<<< 


Figure 47. General Weapon Characteristics 


Figure 48 shows weapon paintball characteristics assigned by the author 
for this scenario. 


Paintball Characteristics 
| rr—“(i‘(SCSC™;™C~*dU:SC(‘(#$NNTC DDeeitd §-_ <$<— $s] —CsCiéieltaae— ‘$e 
Type Red —[oreen "ete — apa Beta [Gara 
i = 





Figure 48. Paintball Weapon Characteristics 


Figure 49 represents probability of hit data entered for the weapons in 
the model. 


Probability of Hit 











Bomb (Carleton) See probability of kill data 
Bomb (Cookie Cutter) i See probability of kill data 
Medical Kit ; R F . 0.00 0.00 0.00 
Orders - From Firemen i . : B 0.50 0.50 0.50 
Orders - From Medics “ . ‘ i 0.50 0.50 0.50 
Orders - From Police . 7 A B 0.50 0.50 0.50 

*Units of measurement for Pythagoras distances are pixels 

“Units of measurment for Real distances are feet 





Figure 49. Probability of Hit Data 


Figure 50 shows the probabilities of kill (for lethal weapons) or 
probabilities of restoration (restorative weapon), given that targets are hit in a 
certain engagement. This is different than SSPK. SSPK is calculated for a 


certain weapon, at a range by multiplying the numbers given in Figures 49 and 


133 


50. If the range is not shown in the figures, use linear interpolation between the 


next highest and next lowest number. 
Probability of Kill Given Hit 
ee 
OS SO CT 
[Real 


Pots a8 6s as 328] a0 492574 | 656 | 








Bomb (Carleton) 

Bomb (Cookie Cutter) 

Medical Kit 

Orders - From Firemen 

Orders - From Medics 

Orders - From Police 
*Units of measurement for Pythagoras distances are pixels 
“Units of measurment for Real distances are feet 





Figure 50. Probability of Kill, Given Hit Data 
4. Agent Side Property 


Figures 51 and 52 provide detailed information about the side affiliations 
of each agent class instantiated in this simulation, using a spreadsheet tool 
developed by Northrup Grumman. 
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Figure 52. Side Affil 


Movement Rates 


5. 


Figure 53 is a screen shot of an Excel spreadsheet movement calculator 


adapted from work done by Babilot.® 
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Excel Movement Rate Calculator 


Figure 53. 


Michael Babilot, “Comparison of a Distributed Operations Force to a Traditional Force in 
Urban Combat,” Masters Thesis, Naval Postgraduate School, Monterey, CA, September 2005. 
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This appendix served to answer questions the reader may have had about 
the numbers used to develop this simulation by describing where the data came 


from and how the numbers were developed. 
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APPENDIX B — EXPERIMENTAL DESIGN METHODOLOGY 


This appendix provides additional detail about the procedures used to 
develop the experimental designs for this research. It expands upon the 
discussion of OLHs, NOLHs, and FRLHs, to provide more information on how the 
number of experimental design points is calculated for a given simulation. In 
addition, this appendix provides the Visual Basic code that can form the bridge 
between any Latin Hypercube builder and study file for Pythagoras or MANA. 


A. DESIGN POINT CALCULATIONS 
1. Design by Ye 
Figure 54 shows the relationship between number of variables and 


required levels to maintain orthogonality of design points, as determined by Ye. 


Step 1: Determine number of variables (k) 


Step 2: Calculate m= <= 


Step 3: Calcuate number of required levels n= 2” +1 





Figure 54. Relationship of k, m, and n, as determined by Ye 
2. Design by Cioppa 


Table 23 illustrates a finding by Cioppa that 


. .. as the number of levels doubles (less one for the center point), 
Ye’s [Orthogonal Latin Hypercube] OLHC designs are able to 
accommodate exactly two more variables.” In the new designs [by 
Cioppa], the corresponding maximum number of variables 
increases in accordance with the following formula.” 





°K.Q. Ye, “Orthogonal Column Latin Hypercubes and their Application in Computer 
Experiments,” Journal of the American Statistical Association — Theory and Methods, Vol. 94, 
No. 144, pp. 14380-1439, December 1998. 


Thomas M. Cioppa, “Efficient Nearly Orthogonal and Space-Filling Experimental Designs 


for High-Dimensional Complex Models,” Ph.D. Dissertation, Operations Research Department, 
Naval Postgraduate School, Monterey, CA, 2002. 
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Figure 55. Determination of Maximum Variables for Given Number of Design Points 
(Cioppa) 


Total Number of Maximum Number of Maximum Number of 
Levels for Each Variables Using Variables Using 
Variable (n) Cioppa’s Design (k) Ye’s Design 





Table 22. A Comparison Illustrating the Increased Number of Variables that can be 
Examined by Extending Ye’s (1998) Construction Algorithm for OLHCs 


3. Design by Hernandez 


Hernandez” found that that as the number of levels increases, the 
possible number of variables that can be explored increases to no more than 
one-third the number of levels. Hernandez found that using correlation reductin 
methods, it is possible to create a design that meets Cioppa’s NOLH criteria, 


when the number of design points is greater than or equal to 49 (see Figure 56). 


number of design points (levels) 
number of variables 


When n= 49, 
n= 3k 





Figure 56. Relationship Between Number of Variables and Number of Levels in 
Hernandez’s Design of Experiments 





“Alejandro D. Hernandez, “Expanding the Family of Efficient, High-Dimensional, Nearly 
Orthogonal Experimental Designs through Flexible Random Latin Hypercubes Methods,” 
Ph.D. Dissertation, Operations Research Department, Naval Postgraduate School, Monterey, CA, 


pending publication. 
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Table 24 highlights the difference between Cioppa’s design and 
Hernandez’s design in the number of possible variable analyzed in a given 
number of possible design points. 


Total Number of Maximum Number of Maximum Number of 
Levels for Each Variables Using Variables Using 
Variable (n) Hernandez’s Design (k) Cioppa’s Design 


po B7 | | HTC 
PTS 


Table 23. A Comparison Illustrating the Increased Number of Variables that can be 
Examined by Extending Cioppa’s Construction Algorithm for NOLHs 





B. EXCEL MACRO BY MICHEL 


Figure 57 is Visual Basic code, developed by Major Christopher Michel, 
USMC.” This code was developed to run in Microsoft Excel; it converts a 
Latin Hypercube into a format that easily imported into a study file to execute a 
data farming experiment in either Pythagoras or MANA. 


“Christopher Michel, “Supporting A Marine Air Ground Task Force With Appropriate 
Quantities of Ground Based Fire Support,’ Masters Thesis, Naval Postgraduate School, 
Monterey, CA, September 2006. 
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Sub TransposeNumberMacro {) 


' TransposeNumberMacro Macro 


Macro recorded 3/27/2006 by localadmincm 


‘Ensure data starts in 2nd column to use this code 
xX = 0 


For DesignPoint = O To 144 


For j = 1 To 48 'factors 
Sheets ("YOUR_LATIN_HYPERCUBE_GOES_HERE") . Select 
Cells(2 + DesignPoint, j + 1) .Copy 
Sheets ("VBA for XML") .Select 
Cells(xX + j + 1, 2) .Select 
ActiveSheet.Paste 
Application.CutCopyMode = False 


Next j 


xX = xX + 50 'transposes data 


Next DesignPoint 
End Sub 











Figure 57. Excel Macro by Michel 
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APPENDIX C — DATA ANALYSIS 


This appendix provides an illustration of the Clementine stream used to 


clean the data resulting from the simulation runs conducted. 


A. LARGE FRLH EXPERIMENT 


Figure 58 Illustrates the process by which Clementine can be used to 
transform the data received from a Pythagoras submission to MHPCC into a 
data set that contains exactly the information required for analysis. 


Large Experiment in Clementine 
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Figure 58. Using Clementine to Clean Large FRLH Data 
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B. SMALL FRLH EXPERIMENT 


The same Clementine stream is used to clean the small data set as was 
used for the large. The input files are simply deleted, and the files corresponding 


to the small FRLH data are uploaded. 


C. GRIDDED EXPERIMENT 


The gridded experiment required the same approach to the FRLH 
experiments, but required the addition of two additional file inputs. The existing 
streams were simply copied and pasted; it was not necessary to build them 


from the start. 
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APPENDIX D — PAIW-12 FINDINGS 


This appendix provides information about findings that resulted from the 
PAIW-12 workshop, held in Boppard, Germany, from June 1-10, 2006. The work 
of PAIW-12 Syndicate 2 was a follow on work from the research presented in this 
document. Syndicate 2 was composed of the following people: 


° Major Jonathan W. Roginski, United States Army 


° Ms. Gudrun Wagner, Germany, European Aeronautic Defence and 
Space (EADS) Systems, Defence and Security 


° Mr. Ole Jakob Sendstad, Norway, Norwegian Defence 
Research Establishment 


The overall goal of the workshop for Syndicate 2 was to model pro and 
anti first responder civilians, and the effect of neutral civilians adopting an 
allegiance of either pro or anti first response, while decreasing simulation run 
time. The syndicate used the following MOEs to evaluate first responder 
success; all statistics were collected at the end of the simulation run. 


° Percentage of civilians killed or injured 


e Percentage of civilians near the triage point 


Figure 59 is an overview of what was accomplished during the week. 
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What was Accomplished 





* Simulation pared down from 40+ minutes to 8-12 minute 
run time 


« 2000+ MHPCC simulation runs to analyze impact of 


civilian numbers 
_., SB variables Flexible Random 
Latin Hypercube 








— 144 design points (15 replicates) 
— MOE 1: Percentage of Civilians Killed or Injured 
— MOE 2: Percentage of Civilians near Triage Point 











Figure 59. PAIW-12 Accomplishments 
The syndicate was able to complete the modeling of pro and anti response 
civilians, in addition to neutral civilian transitions in less than one week, but was 
not able to have the data set returned from MHPCC in time to analyze before the 


close of the workshop. 


A. MOE 1 


The syndicate did receive the base case scenario submission in time for 
analysis, resulting in some interesting results. Figure 60 is an illustration of 


analysis results that focused on MOE 1 from Figure 59. 
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Figure 60. MOE 1 Results 


The CART diagram (not shown) indicated that the two most important 
variables in determining the percentage of civilians killed or injured were the 
number of police at intersection 1 (northwest corner) and the lethality of the 
bomb. The contour plot in Figure 60 plots the two most important variables on 
the horizontal and vertical axes and uses color to emphasize the response 
variable: civilians injured or killed. 

Analysis of MOE 1 seems to indicate that, while the number of police at 
this location is important, more is not always better. In this simulation, there 
appears to be diminishing returns after three police arrive at the location; there is 
little increase in the MOE with additional police at the location. Although a 
low resolution simulation cannot be trusted to determine that the number 3 is 


exactly the correct number at which the diminishing returns happens, it can be an 
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indicator that the point of diminishing returns exists and that the decision maker 
should be aware of it. 


B. MOE 2 


Using a similar technique to that used for MOE 1, the syndicate first used 
a CART diagram to identify the most important variables. For this MOE, the most 
important variables were the lethality of the bomb, and the number of police that 
patrol the middle patrol area. The contour plot in Figure 61 plots the two most 
important variables on the horizontal and vertical axes and uses color to 


emphasize the response variable: civilians injured or killed. 





Contour Plot for Percent_Near_Triage_Point 
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Bomb_Lethalit 
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Figure 61. MOE 2 Results 


Analysis of MOE 1 seems to indicate a similar diminishing return in the 
central patrol area as was indicated at the northwest intersection in MOE 1, 
although the relationship is not as strong. 
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Overall, analysis of the data resulting from PAIW-12 suggests: 


It is possible to create the forces in an emergency response 
simulation, debug the scenario, and use data farming techniques to 
receive data to analyze within one week. 


There may be an optimal number of traffic police per intersection 
and patrolmen per area. Exceeding that number could result in 
diminishing returns, or an inefficient use of manpower. 
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